Smart device

ABSTRACT

An Internet of Thing (IoT) device includes a body with a processor, a camera and a wireless transceiver coupled to the processor.

BACKGROUND

The present invention relates to the Internet of Things (IoT).

SUMMARY

In one aspect, an Internet of Thing (IoT) device includes a processorcoupled to a wireless transceiver; a camera; an accelerometer to detectacceleration of the device; and a module to follow a third-party motionor another device motion based on camera and accelerometer outputs.

These and other features of the present invention will become readilyapparent upon further review of the following specification anddrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates an exemplary environment for communicating data froma monitoring device to external computers

FIG. 1B is a perspective view of an exemplary IoT device system.

FIG. 1C is an exemplary process supported by the device according to thepresent invention.

FIG. 2A is a block diagram of an electronic circuit for a smart device.

FIG. 2B is a block diagram of a big data system for predicting stressexperienced by a structural unit such as a bridge, a building, or aplane, for example.

FIG. 3 is a flowchart illustrating one operation of the system of FIG.2A-2B in detecting stress on a unit.

FIG. 4 shows an exemplary sports diagnosis and trainer system foraugmented and/or virtual reality.

FIG. 5 shows an exemplary process for augmented and/or virtual realityfor viewers participating in a game.

FIG. 6 shows an exemplary process to identify reasons for sensor datachanges using a gaming process.

FIG. 7 shows an exemplary glove, FIG. 8 shows an exemplary smart band,FIG. 9 shows exemplary smart clothing, FIG. 10 shows exemplary smartballs.

FIG. 11A shows exemplary smart rackets while FIG. 11B shows electronicsin the handle for golf clubs, rackets, or kung fu sticks.

FIG. 12A-12B show exemplary protective gears, while FIG. 12C shows anexemplary process to fabricate mass-customized protective gear.

FIG. 13 shows an exemplary smart vehicle.

FIG. 14 shows an exemplary vehicle coordination system.

FIG. 15A shows an exemplary virtual reality camera mounted on a gear,and FIG. 15B shows exemplary augmented reality real-time coaching of aplayer such as a quarterback during fourth down.

FIG. 16A-16C shows exemplary coaching system for skiing, bicycling, andweightlifting/free style exercise, respectively, while FIG. 16D shows akinematic modeling for detecting exercise motion which in turn allowsprecision coaching suggestions.

Similar reference characters denote corresponding features consistentlythroughout the attached drawings.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

According to various embodiments of the present disclosure, anelectronic device may include communication functionality. For example,an electronic device may be a smart phone, a tablet Personal Computer(PC), a mobile phone, a video phone, an e-book reader, a desktop PC, alaptop PC, a netbook PC, a Personal Digital Assistant (PDA), a PortableMultimedia Player (PMP), an MP3 player, a mobile medical device, acamera, a wearable device (e.g., a Head-Mounted Device (HMD), electronicclothes, electronic braces, an electronic necklace, an electronicaccessory, an electronic tattoo, or a smart watch), and/or the like.

According to various embodiments of the present disclosure, anelectronic device may be a smart home appliance with communicationfunctionality. A smart home appliance may be, for example, a television,a Digital Video Disk (DVD) player, an audio, a refrigerator, an airconditioner, a vacuum cleaner, an oven, a microwave oven, a washer, adryer, an air purifier, a set-top box, a TV box (e.g., SamsungHomeSync™, Apple TV™, or Google TV™), a gaming console, an electronicdictionary, an electronic key, a camcorder, an electronic picture frame,and/or the like.

According to various embodiments of the present disclosure, anelectronic device may be a medical device (e.g., Magnetic ResonanceAngiography (MRA) device, a Magnetic Resonance Imaging (MRI) device,Computed Tomography (CT) device, an imaging device, or an ultrasonicdevice), a navigation device, a Global Positioning System (GPS)receiver, an Event Data Recorder (EDR), a Flight Data Recorder (FDR), anautomotive infotainment device, a naval electronic device (e.g., navalnavigation device, gyroscope, or compass), an avionic electronic device,a security device, an industrial or consumer robot, and/or the like.

According to various embodiments of the present disclosure, anelectronic device may be furniture, part of a building/structure, anelectronic board, electronic signature receiving device, a projector,various measuring devices (e.g., water, electricity, gas orelectro-magnetic wave measuring devices), and/or the like that includecommunication functionality.

According to various embodiments of the present disclosure, anelectronic device may be any combination of the foregoing devices. Inaddition, it will be apparent to one having ordinary skill in the artthat an electronic device according to various embodiments of thepresent disclosure is not limited to the foregoing devices.

In one embodiment, a smart device includes sensor(s) and wirelesscommunication therein. The device can detect tension and communicate toa computer for storage and analysis. The smart device provides anautomatic electronic process that eliminates the need for a manualinspection process, and uses electronic detection of stress, eliminatingsubjective human judgments and producing greater uniformity inmaintenance, inspection, and emergency detection procedures.

FIG. 1A illustrates an exemplary environment for communicating data froma monitoring device to external computers. In FIG. 1A, the monitoringdevice used for a device 9 includes an interface with a radiotransmitter for forwarding the result of the comparison to a remotedevice. In one example, the monitoring device may include an additionalswitch and user interface. The user interface may be used by the user inorder to trigger transmission of the comparison of the hand or footpattern reference data with the stroke patterns data to the remotedevice. Alternatively, the transmission may occur automatically eachtime the device has been used, or may be triggered by placing the devicein a cradle or base. All parts of the monitoring device may beencapsulated with each other and/or may be integrated into or attachedto the body of the device 9. Alternatively, a radio transmitter may bearranged separately from the other parts, for instance, in a batterycharger, cradle or base of the device 9. In that example, the interface7 may include contact terminals in the device 9, which are connected tothe corresponding terminals in the battery charger for forwarding theresult of the comparison via a wired connection to the transmitter inthe battery charger or may be connected by induction or short rangewireless communications. The radio transmitter in the battery chargerthen transmits this comparison result further via the wireless radioconnection to the remote device. In FIG. 1A, the remote device may be amobile phone 16, PDA or computer 19, which receives the informationdirectly from the monitoring device via a short range radio connection,as one example of a transmitter, such as a Bluetooth or a Wifi or aZigbee connection. In one example, the user of the remote device mayreceive information about how thoroughly the device 9 has been used orthe need to provide a replacement device. FIG. 1A also illustrates analternate example of a transmitter, using an intermediate receiver 17and a network 18, such as a cellular radio system. Also in this example,the radio transmitter may be located in connection with the device 9 oralternatively in connection, with a charger, cradle or base station ofthe device 9. In such an example, the comparison result may betransmitted via an intermediate receiver 17 and the network 18 to aremote device 19, 16 located further away than the range of a shortrange radio system, for example. The remove device 19, 16 may be anydevice suitable for receiving the signals from the network 18 andproviding feedback on an output device. The transmission of informationvia a cellular radio system to the remote device may allow an advertiserprovide an advertisement. For example, an advertisement may be added tothe comparison result using network elements in the cellular radiosystem. The user may receive an advertisement with the comparisonresult. An advantage with such a solution is that the advertiser mayprovide revenue offsetting all or a portion of the cost for thetransmission of the comparison result from the device 9 to the remotedevice 19, 16.

FIG. 1B shows a block diagram of the unit 9 with processor/RAM/ROM 11.The unit 9 includes a motion sensor, a multi-axis accelerometer, and astrain gage 42. The multi-axis accelerometer may be a two-axis orthree-axis accelerometer. Strain gage 21 is mounted in the neck of theracket, and measures force applied to the ball, i.e., force in a zdirection. Acceleration and force data are acquired by themicroprocessor at a data acquisition rate (sampling rate) of from about10 to 50 samples/second, e.g., about 20 samples/second. The accelerationdata is used to infer motion, using an algorithm discussed below; it isnot converted to position data. In this embodiment, because the sensorsand strain gage are not in the head region, the head can be removableand replaceable, e.g., by threaded engagement with the handle (notshown), so that the device can continue to be used after instrument wearhas occurred. Any desired type of removable head or cartridge can beused.

The unit 11 also includes a camera, which can be a 360 degree camera.Alternatively, the camera can be a 3D camera such as the Kinect cameraor the Intel RealSense camera for ease of generating 3D models and fordetecting distance of objects. To reduce image processing load, eachcamera has a high performance GPU to perform local processing, and theprocessed images, sound, and odor data are uploaded to a cloud storagefor subsequent analysis.

The unit 11 includes an electronic nose to detect odor. The electronicnose can simply be a MEMS device acting as a particle counter. Anembodiment of the electronic nose can be used that includes a fanmodule, a gas molecule sensor module, a control unit and an output unit.The fan module is used to pump air actively to the gas molecule sensormodule. The gas molecule sensor module detects the air pumped into bythe fan module. The gas molecule sensor module at least includes a gasmolecule sensor which is covered with a compound. The compound is usedto combine preset gas molecules. The control unit controls the fanmodule to suck air into the electronic nose device. Then the fan moduletransmits an air current to the gas molecule sensor module to generate adetected data. The output unit calculates the detected data to generatea calculation result and outputs an indicating signal to an operator orcompatible host computer according to the calculation result.

An electronic tongue sensor can be provided to sense quality of sweat orliquid. The tongue includes a liquid molecule sensor module, a controlunit and an output unit. Body liquid is applied or swiped on to theliquid molecule sensor module. The molecule sensor module detects theliquid molecules pumped into by the stirring module. The liquid moleculesensor module at least includes a molecule sensor which is covered witha compound. The compound is used to combine preset liquid molecules. Thecontrol unit controls the stirring module to pump liquid to be “tasted”into the electronic tongue device. Then the module transmits a flowcurrent to the liquid molecule sensor module to generate a detecteddata. The output unit calculates the detected data to generate acalculation result and outputs an indicating signal to an operator orcompatible host computer according to the calculation result. Suchelectronic tongue can detect quality of fog or liquid, among others.

In one embodiment for analyzing tooth structure, restorative materialswithin a tooth structure, and disease states of a tooth, the unit 11includes a probe 20 which may be attached to a variety of probes, andinstruments to afford adaptability to a variety of situations inproviding diagnostic information on an object such as a naturallyoccurring structure, man-made materials placed or found within thestructure, diseased or otherwise affected, infected or effectedstructure, as well as structure that has been eroded, worn by attrition,abraded, abfracted, fractured, crazed, broken or otherwise compromisedthrough enthusiast use, misuse, fatigue or longevity of use. The probe20 generates electrical outputs which are interpreted by a smart phoneor computer.

In one embodiment, the probe 20 can be a vibratory transducer that sendsout vibrations at known frequency and amplitude. The probe 20 alsoincludes a receiver which can be an accelerometer, for example. Theaccelerometer is attached to the teeth and connected to a computer. Theaccelerometer digitizes the received vibrations and provides them intothe phone or computer. The transducer can be a single piezoelectrictransducer or an array with elements arranged to fit in a mouthpiece oran appliance to be worn over the oral arch. The transducer elements canbe mounted in silicone rubber or other material suitable for dampingmechanical coupling between the elements. Other materials may also beused for the array construction. For example, the transducer may beformed from one or more pieces of piezocomposite material, or anymaterial that converts electrical energy to acoustic energy. Thereceiver can also be positioned to fit in the mouthpiece or appliance.One embodiment of the receiver is an accelerometer, but a suitablepiezoelectric transducer can serve as the receiver as well.

The software in the computer compares these inputs to known vibrationresponses corresponding to striking states on a ball or object. Thecomputer 30 displays a response on the computer screen for that user.

FIG. 1C schematically shows a method or app 2 which may be implementedby the computing unit 11 shown in FIG. 1B. For example, the app 2 may bea computer implemented method. A computer program may be provided forexecuting the app 2. The app 2 includes code for:

(21) capture user motion with accelerometer or gyroscope

(22) capture VR views through camera and process using GPU

(23) capture user emotion using facial recognition or GSR

(24) model user action using kinematic model

(25) compare user action with idea action

(26) coach user on improvement to user techniques.

As shown in FIG. 2A, a microcontroller 155 receives and processessignals from the sensor 112-114, and converts those signals into anappropriate digital electronic format. The microcontroller 155wirelessly transmits tension information in the appropriate digitalelectronic format, which may be encoded or encrypted for securecommunications, corresponding to the sensed traffic and/or crimeindication through a wireless communication module or transceiver 160and antenna 170. Optionally, a camera 140 can be provided to visuallydetect traffic and/or crime and movement of the structure. Whilemonitoring of the smart device 100 traffic and/or crime is continuous,transmission of tension information can be continuous, periodic orevent-driven, such as when the tension enters into a warning oremergency level. Typically the indicated tension enters a warning level,then an emergency level as tension drops below the optimal range, butcorresponding warning and emergency levels above the optimal range canalso be used if supported by the smart device 100. The microcontroller155 is programmed with the appropriate warning and emergency levels, aswell as internal damage diagnostics and self-recovery features.

The tension information can take any form, including a simplewarning/emergency indication that the tension is approaching orexceeding tension specifications, respectively. While under-tension isknown to be the primary cause of structural or mechanical problemsassociated with devices, over-tension can also be a problem and can alsobe reported by the smart device 100.

The sensors can detect force, load, tension and compression forces onthe device such as the device. Other data includes Acceleration;Velocity; Global absolute displacement; Local relative displacement;Rotation; Strain; Stress; Force; and Static-position video. Windspeed/direction; External temperature; weather parameters (rainfall,humidity, solar radiation, etc.); Internal or structural temperature;Mass loading (occupant count, etc.); Static tilt; Fatigue damage;Corrosion; Acoustic emission; and Moving-position video. A force issimply a push or pull to an object and can be detected by a load cell,pressure cell or strain sensor. A Load: Is simply a force applied to astructure. Ex: weight of vehicles or pedestrians, weight of wind pushingon sides. Tension & Compression are internal forces that make a memberlonger or shorter. Tension stretches a member and Compression pushes themember closer together. Acceleration can also be detected byForce-Balance (Servo) Piezoelectric Piezoresistive MEMS. Velocity can bemeasured by force-balance (servo) MEMS, or Mechanical Doppler Heatedwire. A local Displacement sensor can be LVDT/Cable potentiometerAcoustic Optical/laser Temperature Electrical Optical fiber. A rotationsensor can be Gyro MEMS Gyro Tilt Electro-mechanical MEMS. A strainsensor can be a resistance gauge Vibrating wire Optical fiber CorrosionElectrical Chemical sensors. A traffic and/or crime sensor can be amicrophone listening to acoustic emission, or Piezoelectric MEMS, forexample, and sonar sound processing can be used to detect where crimeactivity is coming from.

The sensor 112-114, transceiver 160/antenna 170, and microcontroller 155are powered by and suitable power source, which may optionally includean electromagnetic field (EMF) scavenging device 145, such as thoseknown in the art, that convert ambient EMF (such as that emitted byradio station broadcasts) into small amounts of electrical power. TheEMF scavenging device 145 includes a battery to buffer and store energyfor the microcontroller 155, sensor 112-114, camera 140 and wirelesscommunications 160/170, among others.

The circuit of FIG. 2A contains an analog front-end (“AFE”) transducer150 for interfacing signals from the sensor 112-114 to themicrocontroller 155. The AFE 150 electrically conditions the signalscoming from the sensor 112-114 prior to their conversion by themicrocontroller 155 so that the signals are electrically compatible withthe specified input ranges of the microcontroller 155. Themicrocontroller 155 can have a CPU, memory and peripheral circuitry. Themicrocontroller 155 is electrically coupled to a wireless communicationmodule 160 using either a standard or proprietary communicationstandard. Alternatively, the microcontroller 155 can include internallyany or all circuitry of the smart device 100, including the wirelesscommunication module 160. The microcontroller 155 preferably includespower savings or power management circuitry 145 and modes to reducepower consumption significantly when the microcontroller 155 is notactive or is less active. The microcontroller 155 may contain at leastone Analog-to-Digital Converter (ADC) channel for interfacing to the AFE150.

The battery/power management module 145 preferably includes theelectromagnetic field (EMF) scavenging device, but can alternatively runoff of previously stored electrical power from the battery alone. Thebattery/power management module 145 powers all the circuitry in thesmart device 100, including the camera 140, AFE 150, microcontroller155, wireless communication module 160, and antenna 170. Even though thesmart device 100 is preferably powered by continuously harvesting RFenergy, it is beneficial to minimize power consumption. To minimizepower consumption, the various tasks performed by the circuit should berepeated no more often than necessary under the circumstances.

Stress information from the smart device 100 and other information fromthe microcontroller 155 is preferably transmitted wirelessly through awireless communication module 160 and antenna 170. As stated above, thewireless communication component can use standard or proprietarycommunication protocols. Smart lids 100 can also communicate with eachother to relay information about the current status of the structure ormachine and the smart device 100 themselves. In each smart device 100,the transmission of this information may be scheduled to be transmittedperiodically. The smart lid 100 has a data storage medium (memory) tostore data and internal status information, such as power levels, whilethe communication component is in an OFF state between transmissionperiods. On the other hand, once the communication commences in the ONstate, the microcontroller 155 can execute the following tasks:

1. Neighbor discovery: in this task each smart device 100 sends a beaconidentifying its location, capabilities (e.g. residual energy), status.2. Cluster formation: cluster head will be elected based on the findingsin (1). The cluster children communicate directly with their clusterhead (CH). 3. Route discovery: this task interconnects the electedcluster heads together and finds the route towards the sink smart device(node) so that minimum energy is consumed. 4. Data transmission: themicrocontroller processes the collected color data and based on theadopted data dissemination approach, the smart device 100 will do one ofthe following. (a) Transmit the data as is without considering theprevious status; or (b) transmit the data considering the previousstatus. Here we can have several scenarios, which include: (i)transmitting the data if the change in reported tension exceeds thewarning or emergency levels; and (ii) otherwise, do not transmit.

The electronic of FIG. 2A operates with a big data discovery system ofFIG. 2B that determines events that may lead to failure. FIG. 2B is ablock diagram of an example stress monitoring system 200 that may beprocess the stress detected by the smart device 100 of FIG. 1, arrangedin accordance with at least some embodiments described herein. Alongwith the stress monitoring system 220, a first smart device such as asmart device 240, a second smart device 250, a third smart device 260, afourth smart device 280, and additional sensors 270 may also beassociated with the unit 200. The stress monitoring system 220 mayinclude, but is not limited to, a transceiver module 222, a stressdetection module 224, a stress prediction module 226, a determinationmodule 228, a stress response module 232, an interface module 234, aprocessor 236, and a memory 238.

The transceiver module 222 may be configured to receive a stress reportfrom each of the first, second, and third smart devices 240, 250, 260.In some embodiments, the transceiver module 222 may be configured toreceive the stress reports over a wireless network. For example, thetransceiver module 222 and the first, second, and third smart devices240, 250, 260 may be connected over a wireless network using the IEEE802.11 or IEEE 802.15 standards, for example, among potentially otherstandards. Alternately or additionally, the transceiver module 222 andthe first, second, and third smart devices 240, 250, 260 may communicateby sending communications over conductors used to carry electricity tothe first, second, and third smart devices 240, 250, 260 and to otherelectrical devices in the unit 200. The transceiver module 222 may sendthe stress reports from the first, second, and third smart devices 240,250, 260 to the prediction module 226, the stress detection module 224,and/or the determination module 228.

The stress module 224 may be configured to detect stress on the objectas detected by the devices 100. The signal sent by the devices 100collectively may indicate the amount of stress being generated and/or aprediction of the amount of stress that will be generated. The stressdetection module 224 may further be configured to detect a change instress of non-smart devices associated with the unit 200.

The prediction module 226 may be configured to predict future stressbased on past stress history as detected, environmental conditions,forecasted stress loads, among other factors. In some embodiments, theprediction module 226 may predict future stress by building models ofusage and weight being transported. For example, the prediction module226 may build models using machine learning based on support vectormachines, artificial neural networks, or using other types of machinelearning. For example, stress may correlate with the load carried by abridge or an airplane structure. In other example, stress may correlatewith temperature cycling when a structure is exposed to constant changes(such as that of an airplane).

The prediction module 226 may gather data for building the model topredict stress from multiple sources. Some of these sources may include,the first, second, and third smart devices 240, 250, 260; the stressdetection module 224; networks, such as the World Wide Web; theinterface module 234; among other sources. For example, the first,second, and third smart devices 240, 250, 260 may send informationregarding human interactions with the first, second, and third smartdevices 240, 250, 260. The human interactions with the first, second,and third smart devices 240, 250, 260 may indicate a pattern of usagefor the first, second, and third smart devices 240, 250, 260 and/orother human behavior with respect to stress in the unit 200.

In some embodiments, the first, second, and third smart devices 240,250, 260 may perform predictions for their own stress based on historyand send their predicted stress in reports to the transceiver module222. The prediction module 226 may use the stress reports along with thedata of human interactions to predict stress for the system 200.Alternately or additionally, the prediction module 226 may makepredictions of stress for the first, second, and third smart devices240, 250, 260 based on data of human interactions and passed to thetransceiver module 222 from the first, second, and third smart devices240, 250, 260. A discussion of predicting stress for the first, second,and third smart devices 240, 250, 260 is provided below with respect toFIGS. 5 and 6.

The prediction module 224 may predict the stress for different amountsof time. For example, the prediction module 224 may predict stress ofthe system 200 for 1 hour, 2 hours, 12 hours, 1 day, or some otherperiod. The prediction module 224 may also update a prediction at a setinterval or when new data is available that changes the prediction. Theprediction module 224 may send the predicted stress of the system 200 tothe determination module 228. In some embodiments, the predicted stressof the system 200 may contain the entire stress of the system 200 andmay incorporate or be based on stress reports from the first, second,and third smart devices 240, 250, 260. In other embodiments, thepredicted stress of the system 200 may not incorporate or be based onthe stress reports from the first, second, and third smart devices 240,250, 260.

The determination module 228 may be configured to generate a unit stressreport for the system 200. The determination module 228 may use thecurrent stress of the system 200, the predicted stress of the system 200received from the prediction module 224; stress reports from the first,second, and/or third smart devices 240, 250, 260, whether incorporatedin the predicted stress of the system 200 or separate from the predictedstress of the system 200; and an amount of stress generated or thepredicted amount of stress, to generate a unit stress report.

In some embodiments, one or more of the stress reports from the first,second, and/or third smart device 240, 250, 260 may contain anindication of the current operational profile and not stress. In theseand other embodiments, the determination module 228 may be configured todetermine the stress of a smart device for which the stress reportindicates the current operational profile but not the stress. Thedetermination module 228 may include the determined amount of stress forthe smart device in the unit stress report. For example, both the firstand second smart device 240, 250 may send stress report. The stressreport from the first smart device 240 may indicate stress of the firstsmart device 240. The stress report from the second smart device 250 mayindicate the current operational profile but not the stress of thesecond smart device 250. Based on the current operational profile of thesecond smart device 250, the determination module 228 may calculate thestress of the second smart device 250. The determination module 228 maythen generate a unit stress report that contains the stress of both thefirst and second smart devices 240, 250.

In some embodiments, the stress monitoring system 220 may not includethe prediction module 226. In these and other embodiments, thedetermination module 228 may use stress reports from the first, second,and/or third smart devices 240, 250, 260, with the received amount ofstress inferred on non-smart devices, if any, to generate the unitstress report. The determination module 228 may send the unit stressreport to the transceiver module 222.

In some embodiments, the processor 236 may be configured to executecomputer instructions that cause the stress monitoring system 220 toperform the functions and operations described herein. The computerinstructions may be loaded into the memory 238 for execution by theprocessor 236 and/or data generated, received, or operated on duringperformance of the functions and operations described herein may be atleast temporarily stored in the memory 238.

Although the stress monitoring system 220 illustrates various discretecomponents, such as the prediction module 226 and the determinationmodule 228, various components may be divided into additionalcomponents, combined into fewer components, or eliminated, depending onthe desired implementation. In some embodiments, the unit 200 may beassociated with more or less smart devices than the three smart devices240, 250, 260 illustrated in FIG. 2.

FIG. 3 is a flow chart of an example method 300 of monitoring stress ofa or game unit, arranged in accordance with at least some embodimentsdescribed herein. The method 300 may be implemented, in someembodiments, by an stress monitoring system, such as the stressmonitoring system 220 of FIG. 2. For instance, the processor 236 of FIG.2B may be configured to execute computer instructions to performoperations for monitoring stress as represented by one or more of blocks302, 304, 306, 310, 312, and/or 314 of the method 300. Althoughillustrated as discrete blocks, various blocks may be divided intoadditional blocks, combined into fewer blocks, or eliminated, dependingon the desired implementation.

The method 300 may begin at one or more of blocks 302, 304, and/or 306.The blocks 302, 304, and/or 306 may occur at the same time or atdifferent times and may or may not depend on one another. Furthermore,one or more of the block 302, 304, 306 may occur during the method 300.For example, the method 300 may complete when blocks 304, 310, and 312occurs and without the occurrence of block 302 and 306.

In block 302, a change in stress of a device (device or beam) associatedwith a unit may be detected. A non-smart device may by any device thatreceives stress and does not generate an stress report indicating itsstress, for example a legacy racket without IoT electronics. A change inthe stress of a non-smart device may be detected using an stressdetection module and/or usage meter associated with the unit, such asthe stress detection module 224 and/or the smart device 100. Forexample, non-smart device stress can be estimated by the load the unitcarries, the temperature cycling experienced by the unit, for example.

After a change in stress of the non-smart device is detected, the method300 proceeds to block 310. In block 304, an stress report from a smartdevice such as the smart device 100 associated with the unit may bereceived. A smart device may be a device that detects stress andgenerates and transmits an stress report indicating the stress on thesmart device. The stress report may indicate predicted future stress ofthe smart device. In some embodiments, an stress report may be receivedat set intervals from the smart device regardless of a change in thestress report. Alternately or additionally, a stress report may bereceived after a change in the stress of the smart device results in achange to the stress report. After a stress report is received from thesmart device, the method 300 proceeds to block 310.

In block 306, stress experienced at the unit may be detected. Stress atthe unit may be detected using a stress detection module, such as thestress detection module 224 of FIG. 2B. After detecting stress at theunit, the method proceeds to block 310. At block 310, it is determinedif a change in the stress occurred. For example, if an increase instress occurs at the same time and at the same amount as an increase inthe stress of a non-smart device, a change in the stress may not occur.If a change in the stress occurs, the method 300 proceeds to block 312.If no change occurs, the method 300 ends.

At block 312, a unit stress report is generated for the unit. In someembodiments, the unit stress report may indicate the current stress ofthe unit. Alternately or additionally, the unit stress report mayindicate a current and predicted future stress of the unit. At block314, the unit stress report is transmitted to a maintenance provider. Insome embodiments, the unit stress report may be transmitted when theunit stress report indicates a change in stress for the unit that isgreater than a predetermined threshold. If the unit stress reportindicates a change in stress for the unit that is less than thepredetermined threshold, the unit stress report may not be transmittedto the provider of maintenance services.

FIG. 5 shows in more details the computer 30 and the interface to theprobe 20. An amplifier 90 amplifies vibratory output from a transducer92. A pick up unit having an accelerometer (or an array) 96 receivesreflected vibrations from user arm or leg 94, among others. A computer98 includes a digital converter to digitize output from the pick-up unitand software on the computer 98 can process the captured diagnosticdata. Diagnostic software 100 can include a database of knownrestorations, diseases, and tissue conditions whose signatures can bematched against the capture diagnostic data, and the result can bedisplayed on a screen for review by the athlete.

Included in one embodiment of the instrumentation is the transmitter ortransducer, which will emit the vibrations that will be imparted to theteeth and jaws. This will be connected to a power supply and amplifier,which will allow for a frequency range. On electrical excitation, thetransducer emits an outgoing vibration. That vibration will then travelinto the arm or leg and down is root into the soft tissues and out intothe bones or jaws. The accelerometer or detector will be placed on thebone of interest. It will receive the vibrations from the emitter. Theeffect of the vibrations on the muscle of interest will generate apattern of frequency vibrations. Those vibrations will be digitallyconverted and analyzed against known dental states in the software ofthe computer. As the data is collected various linear samplings andcomparisons will be made against the database. Software will make thesecomparisons as the data is received from the teeth.

FIG. 5 schematically shows a method or app 52 to perform collaborativeVR/AR gaming. The app 52 includes code for:

(51) capture 360 degree view of the live event

(52) detect head position of the viewer

(53) adjust viewing angle on screen based on head position and userposture

(54) render view to simulate action based on user control rather thanwhat the professional is doing

(55) augment view with a simulated object that is powered by vieweraction as detected by sensors on viewer body

(56) compare professional result with simulated result and show resultto a crowd of enthusiasts for social discussion.

FIG. 6 is a flowchart of a method of an embodiment of the presentdisclosure. Referring to FIG. 6, a smart system may collect from smartdevices state change events of a smart system in operation 601. That is,the smart system of FIG. 4 collects information on each of the group ofdevices, the smart devices, the smart appliances, the security devices,the lighting devices, the energy devices, and the like. The state changeevents indicate when there is a change in the state of the device or thesurrounding environment. The state change events are stored by the smartsystem. In operation 603, the system may determine whether a series ofthe collected state change events are a known pattern. That is, thegateway determines whether there are events which have been correlatedor identified in the past. If the collected state change events havebeen identified in the past, it may be necessary to determine that thesmart systems trusts the identification the collected state changeevents. The trust factor of the identification of the collected statechange events may be determined by the number of users who haveidentified the collected state change events or the number of timecollected state change events have been repeated and identified. Inoperation 605, when the series of the collected state change events isan unknown pattern, request users of the smart system to identify whatcaused the collected state change events request. That is, the systemtransmits to a gamification application (hereinafter app) on the user'smobile device a request to identify the collected state change events.The gamification app displays the information and request the user enterinformation identifying the collected state change events. Each of themobile devices transmits this information back to the system to thegamification module. In operation 605, the system transmits the eachuser's identified collected state change events to the other user's ofthe smart home system and they each vote on the best identification ofthe collected state change events. Thus, the identified collected changestate events that have been repeatedly identified over a period of weeksincreases, the trustworthiness of the identification increases.Likewise, if every user of the smart system makes the sameidentification of the collected change state events, the identifiedcollected change state events may be considered trustworthy at point.Such a determination of a threshold for when the identified collectedchange state events are considered trustworthy and therefore need not berepeated, is made by a system administrator. However, it will beunderstood that such a trustworthiness of this type only gives higherconfidence of this particular dataset at that point in time. As suchfurther repetition is required, since the sensor data may have noise,the more datasets to be identified to the pattern, the more robust thetrustworthiness will be. Until the robustness reaches a threshold, thenthe system can confirm this is a known trustworthy pattern.

The system can use gaming to help enthusiasts improve dental care ormaintain teeth hygiene. This may involve use of virtual tools,corresponding to such tools used in normal dental hygiene: device, toothpicks, dental floss, gum massaging aids, etc. In this embodiment, thegame may, for example, have the object of fighting tooth or gum decay,damage or infection which may be caused by carries or other infectiousagents. The user is presented with a library of tools and has to selecta tool to treat a certain developing virtual condition, e.g. carries ora gum infection. The game rules determine a certain continuous progressof infection which if not properly “treated” by the user will causedecay of one or more teeth, gum infection, potential bleeding, loss ofteeth, etc. In step 13, the user may score points depending on hisability to choose the right tools to treat a particular condition or inavoiding a condition from developing. Next, it is determined whether thecondition of the teeth is satisfactory. If yes, the process terminates.If no, then the user is prompted whether he wishes to select anothertool. If no, the process terminates. If yes, the process restarts. Hereagain, the game, in addition to being amusing and providing an insightof the user into his own teeth, may be educational, particularly forchildren, on teeth oral hygiene methods and on the importance ofmaintaining oral hygiene.

In accordance with another embodiment of the invention the game mayinvolve use of a variety of virtual imaginary tools such as virtualguns, wands, etc. in order to fight infectious agents of the teeth orgums.

Smart Glove

FIG. 7 shows an exemplary glove which can be thin to provide touchsensitivity or thick to provide shock protection for boxers. A body 12of the boxing glove 10 includes an impact measuring device 14 isembedded within the glove 12 in an area protected from direct impact.Such an area includes the cuff 15 of the glove 12 or that portion of theglove 12 adjacent a user's palm, or adjacent an inside surface of auser's fingers. Placement of the impact measuring device 14 into thelining of the glove in such an area allows for the force of a blow to bemeasured without presenting a hazard to the recipient of the blow. Underthe embodiment, an impact measuring device 14 would be included in theright glove 12 for a right handed fighter, or the left glove 12 for aleft handed fighter. For fighters that are equally effective with bothhands, or to improve monitoring accuracy, an impact measuring device 14would be included in both gloves 12. The impact measuring system 20. Theimpact measuring system 20 includes an impact measuring device 14 andimpact display unit 16. The impact measuring device 14 is linked to theimpact display 28 via a radio frequency (rf) link 32. Under theembodiment, the impact measuring device 14 includes at least one 3-axisaccelerometer. A thin version of the glove can be worn to detect a golfstroke or a tennis stroke with legacy clubs or rackets that lacks IoTintelligence.

1. A glove comprising:

a glove body;

a processor in the glove body and coupled to a wireless transceiver;

a camera coupled to the glove body;

a sensor disposed in the glove body; and an accelerometer disposedwithin the glove body to detect acceleration of the glove.

2. The glove of claim 1, at least one sensor selected from a sensor setcomprising: a pressure sensor configured to detect at least one pressureevent at a glove body external surface location; a glove motion sensorconfigured to detect at least one motion event of the glove; a digitmotion sensor configured to detect at least one motion event of at leastone digit of the user; a temperature sensor configured to detect atemperature at a glove body external surface location; and a contactsensor configured to detect a contact event of the glove with a contactobject.

3. The glove of claim 1, the sensor comprising at least one of the sameas the second sensor or different than the second sensor, the handexercise event at least one of the same as the second hand exerciseevent or different than the second hand exercise event.

4. The glove of claim 1, the hand exercise regimen selected from a handexercise regimen set comprising at least one of: a physical therapy handexercise regimen; a physical training hand exercise regimen; or aphysical performance hand exercise regimen.

5. The glove of claim 1, the glove comprising a gesture identifyingcomponent configured to identify at least one hand gesture detected byat least one sensor; the memory configured to, upon receiving anindication of a hand gesture identified by the gesture identifyingcomponent, store data corresponding to the hand gesture in the memory;and the device interface configured to, upon connecting to the device,provide at least some of the stored data corresponding to the handgesture to the device.

6. The glove of claim 1, the glove comprising a plurality of fingerreceptacles, each having a sensor.

7. The glove of claim 1, comprising a sensor worn by an opponent inwireless communication with the processor to communicate the force of animpact from the glove.

8. A system for measuring a force of impact of a boxing glove of a boxercomprising:

an accelerometer disposed in the boxing glove of the boxer for measuringthe force of impact of the boxing glove on an opponent;

a radio frequency transmitter disposed in the boxing glove and coupledto the accelerometer for transmitting impact measurements;

a radio frequency receiver for receiving the impact measurements; and

a display coupled to the radio frequency receiver for displaying themeasured impacts.

Smart Band

FIG. 8 shows an exemplary stick on wearable monitoring device for sportsand fitness applications. The wireless sensor electronics 14 is mountedon a band-aid in the example of FIG. 8. The band-aid can be removed uponcompletion of the sports event. The central patch can be recycled, andthe adhesive portion can be disposed. While the embodiment is shown as aband-aid, the inventors contemplate that any suitable bands, straps,attachments can be used in lieu of the band-aid to attach the sensors tothe body. For example, in Virtual Reality (VR) sports applications,sensors including gyroscopes and cameras can be positioned on variousbody portions to capture motion as well as eye tracking, mouth tracking,speech recognition, among others.

One embodiment uses Samsung's Bio-Processor which is an all-in-onehealth solution chip. By integrating not only Analog Front Ends (AFE),but also microcontroller unit (MCU), power management integrated circuit(PMIC), digital signal processor (DSP), and eFlash memory, it is able toprocess the bio-signals it measures without the need of externalprocessing parts. Even with its integrated design, the Bio-Processor isparticularly innovative thanks to its incredibly small size. Whencompared to the total area of the discrete parts, the Bio-Processor isonly about one fourth of the total combined size, which is ideal forsmall wearable devices, offering a bounty of options when designing newdevices. The Bio-Processor has five AFEs including bioelectricalimpedance analysis (BIA), photoplethysmogram (PPG), electrocardiogram(ECG), skin temperature, and galvanic skin response (GSR) into a singlechip solution that measures body fat, and skeletal muscle mass, heartrate, heart rhythm, skin temperature and stress level, respectively.

One embodiment provides a flexible and stretchable electronic patch thatmonitors impact or other events whereby a flexible substrate isgeometrically patterned to allow the substrate to undergo substantialstretching and flexing while large regions of the substrate materialexperiences local strains much lower than the macroscopic appliedstrain. The geometric patterning of the substrate facilitates continuouslow strain domains (LSDs) throughout the substrate—where low straindomains are defined as regions that experience strain levels (magnitude)lower than the macroscopic applied strain. Conventional electroniccomponents can be mounted to the LSDs, and conventional metal traces canbe routed through the LSDs, dramatically reducing the stressestransmitted to the components and traces by the substrate duringstretching and flexing, and therefore reducing the potential forcomponent debonding, trace cracking, and circuit failure. Thegeometrically patterned strain relief features (SRFs) are dispersedeither regularly or irregularly throughout the substrate. Thegeometrically patterned SRF regions form “hinge-like” domains. Duringmacroscopic deformation, the SRFs rotate, translate, open, close, orotherwise change shape, causing the “hinge-like” regions to deform, andthe remaining larger LSD substrate regions to primarily rotate andtranslate. The SRFs are designed such that the “hinge-like” regions alsoexhibit relatively small strain compared to the macroscopic appliedstrain and thus enable conductive traces, such as copper or gold, to runthrough the hinges and maintain function during stretching, flexing andtwisting of the patch. The substrate can be multilayered to enablerunning conductive traces, ground layers, vias, and/or components on/inmultiple layers through the thickness of the overall substrate. Thegeometric patterning can be designed to enable different stretching,flexing and twisting, providing uniaxial, biaxial, and multi-axialstretchability or flexibility, and the ability to conform to a varietyof surface curvatures. The geometrically patterned substrate offers ameans of packaging complex multi-layered electronics designs formonitoring impact (and other) events onto a stretchable and flexiblesubstrate enabling the device to dynamically stretch, bend, twist, andconform to arbitrary shapes. The stretchable, flexible geometricallystructure electronics can be fabricated using the same technologies forconventional flexible circuit boards where the stretch-enablingpatterning can be imparted at different stages in the fabricationprocess and can also be fabricated using emerging materials andfabrication methods. The Stretchable bandaid has the stretchable,flexible substrate described above with multiple LSDs for placement ofelectronic components (e.g., accelerometers, gyroscopes, pressuretemperature, gas and fluid sensors, microprocessors, transceivers, GPS,clocks, actuators, vias, and batteries (or other energy source)) andmultiple patterned hinge-like regions bridging the LSDs which enable therouting of conducting interconnecting traces. The SEHIM patch can takethe form factor of a bandaid or bandage or other such wearable formfactor. The geometric patterning provides stretch, flex and twist toconform to a body and stretch, flex and twist to move or deform with abody. The bandaid detects impact accelerations, using a 3-axisaccelerometer and processes the raw acceleration data in themicroprocessor. The processed data is stored in the microprocessor andlater (or potentially in real time) transmitted via the Bluetooth to asmart phone, tablet or computer. This embodiment encompasses wirelesscommunication but wired communication may be desirable in someapplications and can be accommodated by this invention. The bandaid canbe stretched, bent and twisted with the traces and components at lowstrains to maintain electrical function. In all cases there waseffectively no strain on the components and solder joints. The bandaidcan also possess an adhesive backing for direct adhesion to the head,body or object. The band can also be coated to provide both addedcomfort and protection against moisture, water, and other environmentalfactors. The band can also contain other sensors including gyroscopes,temperature and pressure sensors, moisture sensors, clocks, chemicaland/or biological sensors, etc. Features of the smart band can include:

-   -   1. A smart patch, comprising:        -   a band to be placed over a body portion;        -   a processor in the band and coupled to a wireless            transceiver;        -   a camera coupled to the band;        -   a sensor disposed in the band; and        -   an accelerometer disposed within the band to detect            acceleration of the band.    -   2. The patch of claim 1, comprising a plurality of smart patches        forming a mesh network and communicating episodically to        conserve power.    -   3. The patch of claim 1 where the electronic components,        sensors, and interconnects of the patch monitor, record, process        and/or transmit events of interest (such as accelerometers and        gyroscopes for impact events, temperature sensors for        temperature and/or temperature gradients, pressure sensors,        moisture sensors, chemical sensors).    -   4. The patch of claim 1 comprised for sensing and/or monitoring        impact events where the sensors are accelerometers, gyroscopes,        and/or pressure sensors.    -   5. The patch of claim 1 comprised for sensing and/or monitoring        and/or controlling ongoing events where the sensors monitor        temperature, temperature gradients, motion, position,        environmental or chemical levels, or other such information.    -   6. The patch of claim 1 comprised for sensing events or other        information including mounting multiple distributed sensors for        obtaining spatial and/or temporal distribution in the data        and/or multiple sensors sensing different information and data.    -   7. The patch of claim 1 including wired or wireless        communication, such as a Bluetooth module or a wi-fi module or        other transmission module, transmitting and/or receiving        information to/from another device.    -   8. The patch of claim 1 with power and energy sources including        batteries, wired or wireless rechargeable batteries,        photovoltaics, thermoelectrics, or energy harvesters.    -   9. The patch of claim 1 with an adhesive backing for directly        adhering to a head, a body, or an object.    -   10. The patch of claim 1 contained in an adhesive or a sleeve        for adhering or attaching to a head, a body, or an object.    -   11. The patch of claim 1 coated with a coating for protection        against the elements (water, moisture, dirt, other) and/or for        increased comfort to the wearer.    -   12. The patch of claim 1, comprising a geometrically patterned        substrate that contains regions of low strain domains (LSDs)        bridged by hingeable strain relief features (SRFs) which also        contain low strain regions and enable the stretching, flexing        and twisting of the patch while maintaining continuous low        strain regions for mounting electronic components and routing        traces.    -   13. The patch of claim 1 for attachment to or on or an object,        or embedded in an object.    -   14. The patch of claim 1 in the form factor of a rectangular or        a square or a triangular or other polygon or circular or        elliptical or other geometric shape bandage.    -   15. The patch of claim 1 in the form factor that is or contains        any combination of rectangles, triangles, circles, ellipses or        other form factors.    -   16. The patch of claim 1 with different geometric patterning of        different numbers and shapes and orientations of low strain        domains, different numbers and orientation of geometrically        structured hinge-like domains, and different geometries of        hinge-like domains.    -   17. The patch of claim 1 as a programmable circuit board for        arbitrary applications.    -   18. The patch of claim 1 fabricated using current flex circuit        manufacturing methods and materials.    -   19. The patch of claim 1 comprising a cloud storage to receive        sensor data.    -   20. The patch of claim 1 where the polymer layers are current        flex manufacturing polymers such as Kapton, polyimides,        polyamides, polyesters, or other as well as elastomers such as        silicone rubbers (PDMS) or polyurethanes or other elastomers and        the interconnects are metals that have high electrical        conductivity, such as copper or gold, or where the interconnects        are emerging stretchable electronic materials and stretchable        conductive inks and materials.

Smart Clothing

FIG. 9 shows an exemplary shirt based embodiment where sensors can bepositioned anywhere on the shirt and when worn, can capture position,video, and vital signs. One embodiment uses Samsung's Bio-Processor toprocess the bio-signals it measures without the need of externalprocessing parts with five AFEs including bioelectrical impedanceanalysis (BIA), photoplethysmogram (PPG), electrocardiogram (ECG), skintemperature, and galvanic skin response (GSR) into a single chipsolution that measures body fat, and skeletal muscle mass, heart rate,heart rhythm, skin temperature and stress level, respectively. Featuresof the smart clothe can include:

-   -   1. A smart clothing, comprising:        -   a shirt, underwear, pant or sock;        -   a band to be secured to the a shirt, underwear, pant or            sock;        -   a processor in the band and coupled to a wireless            transceiver;    -   an EKG amplifier coupled to the band;    -   a sensor disposed in the band; and    -   an accelerometer disposed within the band to detect acceleration        of the band.    -   2. The clothing of claim 1, comprising a plurality of bands        forming a mesh network and communicating episodically to        conserve power.    -   3. The clothing of claim 1 where the electronic components,        sensors, and interconnects of the patch monitor, record, process        and/or transmit events of interest (such as accelerometers and        gyroscopes for impact events, temperature sensors for        temperature and/or temperature gradients, pressure sensors,        moisture sensors, chemical sensors).    -   4. The clothing of claim 1 comprised for sensing and/or        monitoring impact events where the sensors are accelerometers,        gyroscopes, and/or pressure sensors.    -   5. The clothing of claim 1 comprised for sensing and/or        monitoring and/or controlling ongoing events where the sensors        monitor temperature, temperature gradients, motion, position,        environmental or chemical levels, or other such information.    -   6. The clothing of claim 1 comprised for sensing events or other        information including mounting multiple distributed sensors for        obtaining spatial and/or temporal distribution in the data        and/or multiple sensors sensing different information and data.    -   7. The clothing of claim 1 including wired or wireless        communication, such as a Bluetooth module or a wi-fi module or        other transmission module, transmitting and/or receiving        information to/from another device.    -   8. The clothing of claim 1 with power and energy sources        including batteries, wired or wireless rechargeable batteries,        photovoltaics, thermoelectrics, or energy harvesters.    -   9. The clothing of claim 1 with an adhesive backing for directly        adhering to a head, a body, or an object.    -   10. The clothing of claim 1 contained in an adhesive or a sleeve        for adhering or attaching to a head, a body, or an object.    -   11. The clothing of claim 1 coated with a coating for protection        against the elements (water, moisture, dirt, other) and/or for        increased comfort to the wearer.    -   12. The clothing of claim 1, comprising a geometrically        patterned substrate that contains regions of low strain domains        (LSDs) bridged by hingeable strain relief features (SRFs) which        also contain low strain regions and enable the stretching,        flexing and twisting of the patch while maintaining continuous        low strain regions for mounting electronic components and        routing traces.    -   13. The clothing of claim 1 for attachment to or on or an        object, or embedded in an object.    -   14. The clothing of claim 1 in the form factor of a rectangular        or a square or a triangular or other polygon or circular or        elliptical or other geometric shape bandage.    -   15. The clothing of claim 1 in the form factor that is or        contains any combination of rectangles, triangles, circles,        ellipses or other form factors.    -   16. The clothing of claim 1 with different geometric patterning        of different numbers and shapes and orientations of low strain        domains, different numbers and orientation of geometrically        structured hinge-like domains, and different geometries of        hinge-like domains.    -   17. The clothing of claim 1 as a programmable circuit board for        arbitrary applications.    -   18. The clothing of claim 1 fabricated using current flex        circuit manufacturing methods and materials.    -   19. The clothing of claim 1 comprising a cloud storage to        receive sensor data.    -   20. The clothing of claim 1 where the polymer layers are current        flex manufacturing polymers such as Kapton, polyimides,        polyamides, polyesters, or other as well as elastomers such as        silicone rubbers (PDMS) or polyurethanes or other elastomers and        the interconnects are metals that have high electrical        conductivity, such as copper or gold, or where the interconnects        are emerging stretchable electronic materials and stretchable        conductive inks and materials.

Smart Handle

FIGS. 11A-11B show an exemplary smart handle for sports such as tennis,badminton, table tennis, and golf, among others. The wireless sensorelectronics 14 is mounted on a handle in the example of FIG. 11B. Thehandle can be embedded or can be removed upon completion of the sportsevent. The sports event does not have to be real, for example, inVirtual Reality (VR) sports applications, sensors including gyroscopesand cameras can be positioned on various body portions to capture motionas well as eye tracking, mouth tracking, speech recognition, amongothers.

The handle includes a swing analyzer measurement portion 54 in the gripend 52 of the handle of a golf club or a tennis/badminton racket, and aremote or handheld unit 56. The swing analyzer measurement portion 54includes an accelerometer 16 of combination accelerometer and gyroscopeor magnetometer unit, a processor unit 58 coupled to the accelerometer16, and a battery 20 that is electrically coupled to and provides powerto the accelerometer 16 and processor unit 58. A camera is included tocapture videos of the swing and also the game in progress for futurereference. A communications unit 60 is also housed in the grip end 52 ofthe golf club 50, receives power from the battery 20, and is coupled tothe processor unit 58. Swing analyzer measurement portion 54, with orwithout the communications unit 60, may be assembled as an integral unitand inserted into a hollow portion of the handle of the golf club ortennis/racket handle 50 at the grip end 52 thereof. Processor unit 58may be an integrated device that includes hardware and softwarecomponents capable of processing acceleration measured by theaccelerometer(s) 16 and converting the measured acceleration into dataabout the force on the shaft and position of the face of the club atimpact at a set distance. If the measured force exceeds a threshold themeasured force or a signal derived therefrom is transmitted via thecommunications unit 60 to the handheld unit 56. If not, acceleration andface position at impact of the golf club or tennis racket handle 50 isobtained again. The threshold is set so that only acceleration or forcemeasurements arising from actual swings of the golf club 50 aretransmitted to the handheld unit 56. Handheld or remote unit 56 includesan application or computer program embodied on a non-transitorycomputer-readable medium that performs the golf ball carrying distanceestimation or prediction steps, as well as manages the training stagedescribed above. Importantly, the handheld unit 56 receives accelerationmeasurement data from the golf clubs/tennis rackets equipped with aswing analyzer measurement portion 54 and the club face angle inrelation to the swing plane, and manages the carrying distanceestimation steps for all golf clubs equipped with the swing analyzermeasurement portion 54 that are designed to communicate therewith.Handheld or remote unit 56 may be a standalone unit for use only withthe golf clubs equipped with the swing analyzer measurement portion 54,and incorporating the application thereon, or may be a smartphone orsimilar device with the application embodied thereon or downloadedthereto and that can be used for other purposes. Handheld or remote unit56 includes a communications unit 70 that communicates with thecommunications unit 60 on each golf club or tennis racket handle 50,i.e., with the communications units present on all of the golf clubs 50equipped with swing analyzer measurement portions 54 and which have beendesignated to communicate therewith. Communications unit 70 may be anintegral part of the handheld unit 56 as is the case when the handheldunit 56 is a smartphone. Communications unit 70 may also communicatewith another device such as a Smartphone, to perform more datamanipulations relating to the golf swing and/or swing results to providemore information to the user. The data and the calculation/manipulationresults can be stored in the Smartphone and displayed when desired.Currently usable Smartphones are Apple iOS iPhones and Android operatingsystem phones. Handheld or remote unit 56 also includes a processor unit72, a storage unit 74 and a display 76. When the handheld unit 56 is asmartphone or similar device, all of the processor unit 72, storage unit74 and display 76 may be integral components thereof. Processor unit 72performs functions similar to those performed by the processor unit 18described above, e.g., calculates an estimated carrying distance for thegolf ball based on the acceleration measured by the accelerometer(s) 16and transmitted via the communications units 60, 70, and the type ofclub provided to the application or computer program in the processorunit 72. Storage unit 74 receives and stores information about thecarrying distance of each club as a function of clock or swing position,e.g., in the form of a virtual table associating the type of club, theswing or swing position and the estimated carrying distance.

Other sensors can be used as well. For example, the handle can containconductive ink to capture biometric. One embodiment uses Samsung'sBio-Processor which is an all-in-one health solution chip to measurebioelectrical impedance analysis (BIA), photoplethysmogram (PPG),electrocardiogram (ECG), skin temperature, and galvanic skin response(GSR) into a single chip solution that measures body fat, and skeletalmuscle mass, heart rate, heart rhythm, skin temperature and stresslevel, respectively. The handle can also contain other sensors includinggyroscopes, temperature and pressure sensors, moisture sensors, clocks,chemical and/or biological sensors, etc. Features of the smart handlecan include:

-   -   1. A smart handle, comprising:        -   a handle;        -   a processor in the band and coupled to a wireless            transceiver;        -   a camera coupled to the handle;        -   a sensor disposed in the handle; and        -   an accelerometer disposed within the band to detect            acceleration of the handle.    -   2. The handle of claim 1, comprising a plurality of smart        handles forming a mesh network and communicating episodically to        conserve power.    -   3. The handle of claim 1 where the electronic components,        sensors, and interconnects of the handle monitor, record,        process and/or transmit events of interest (such as        accelerometers and gyroscopes for impact events, temperature        sensors for temperature and/or temperature gradients, pressure        sensors, moisture sensors, chemical sensors).    -   4. The handle of claim 1 comprised for sensing and/or monitoring        impact events where the sensors are accelerometers, gyroscopes,        and/or pressure sensors.    -   5. The handle of claim 1 comprised for sensing and/or monitoring        and/or controlling ongoing events where the sensors monitor        temperature, temperature gradients, motion, position,        environmental or chemical levels, or other such information.    -   6. The handle of claim 1 comprised for sensing events or other        information including mounting multiple distributed sensors for        obtaining spatial and/or temporal distribution in the data        and/or multiple sensors sensing different information and data.    -   7. The handle of claim 1 including wired or wireless        communication, such as a Bluetooth module or a wi-fi module or        other transmission module, transmitting and/or receiving        information to/from another device.    -   8. The handle of claim 1 with power and energy sources including        batteries, wired or wireless rechargeable batteries,        photovoltaics, thermoelectrics, or energy harvesters.    -   9. The handle of claim 1 with an adhesive backing for directly        adhering to a head, a body, or an object.    -   10. The handle of claim 1 contained in an adhesive or a sleeve        for adhering or attaching to a head, a body, or an object.    -   11. The handle of claim 1 coated with a coating for protection        against the elements (water, moisture, dirt, other) and/or for        increased comfort to the wearer.    -   12. The handle of claim 1, comprising a geometrically patterned        substrate that contains regions of low strain domains (LSDs)        bridged by hingeable strain relief features (SRFs) which also        contain low strain regions and enable the stretching, flexing        and twisting of the handle while maintaining continuous low        strain regions for mounting electronic components and routing        traces.    -   13. The handle of claim 1 for attachment to or on or an object,        or embedded in an object.    -   14. The handle of claim 1 in the form factor of a rectangular or        a square or a triangular or other polygon or circular or        elliptical or other geometric shape bandage.    -   15. The handle of claim 1 in the form factor that is or contains        any combination of rectangles, triangles, circles, ellipses or        other form factors.    -   16. The handle of claim 1 with different geometric patterning of        different numbers and shapes and orientations of low strain        domains, different numbers and orientation of geometrically        structured hinge-like domains, and different geometries of        hinge-like domains.    -   17. The handle of claim 1 as a programmable circuit board for        arbitrary applications.    -   18. The handle of claim 1 fabricated using current flex circuit        manufacturing methods and materials.    -   19. The handle of claim 1 comprising a cloud storage to receive        sensor data.    -   20. The handle of claim 1 where the polymer layers are current        flex manufacturing polymers such as Kapton, polyimides,        polyamides, polyesters, or other as well as elastomers such as        silicone rubbers (PDMS) or polyurethanes or other elastomers and        the interconnects are metals that have high electrical        conductivity, such as copper or gold, or where the interconnects        are emerging stretchable electronic materials and stretchable        conductive inks and materials.

Smart Protective Gear

FIGS. 12A-12C illustrate smart protective gears embedded with the IoTsensors and instrumentations to report potential health issues. Forsoccer, the protection includes shin guards. For football, theprotection includes Helmets, Chin Straps & Chin Shields, Cups & AthleticSupporters, Elbow Sleeves & Arm Pads, Back Plates & Rib Protection,Facemasks, Girdles, Helmet Visors, Shoulder Pads, Hip & Tail Pads,Mouthguards, Neck Rolls. For motorcycling, the protection includeshelmet, should pads, jacket with back protection, padded gloves, leatherpants, knee pads, and boots. For rock climbing, the protection includesshoes, carabiners, webbing, harnesses, among others.

The wireless sensor electronics 14 is mounted on the helmet or shoulderpad in the example of FIG. 12A or 12C. The electronics 14 can beembedded or can be removed upon completion of the sports event. Thesports event does not have to be real, for example, in Virtual Reality(VR) sports applications, sensors including gyroscopes and cameras canbe positioned on various body portions to capture motion as well as eyetracking, mouth tracking, speech recognition, among others.

The protection gear includes an impact sensor such as an accelerometerto indicate if concussion has occurred. Other sensors can be used aswell. For example, the handle can contain conductive ink to capturebiometric. One embodiment uses Samsung's Bio-Processor which is anall-in-one health solution chip to measure bioelectrical impedanceanalysis (BIA), photoplethysmogram (PPG), electrocardiogram (ECG), skintemperature, and galvanic skin response (GSR) into a single chipsolution that measures body fat, and skeletal muscle mass, heart rate,heart rhythm, skin temperature and stress level, respectively. Thehandle can also contain other sensors including gyroscopes, temperatureand pressure sensors, moisture sensors, clocks, chemical and/orbiological sensors, etc.

Impact sensors, or accelerometers, measure in real time the force andeven the number of impacts that players sustain. Data collected is sentwirelessly via Bluetooth to a dedicated monitor on the sidelines, whilethe impact prompts a visual light or audio alert to signal players,coaches, officials, and the training or medical staff of the team. Onesuch sensor example is the ADXL377 from Analog Devices, a small, thinand low-power 3-axis accelerometer that measures acceleration frommotion, shock, or vibration. It features a full-scale range of ±200 g,which would encompass the full range of impact acceleration in sports,which typically does not exceed 150 g's. Specifically designed forconcussion and head-trauma detection, at 3 mm×3 mm×1.45 mm, the deviceis small enough to be designed into a helmet. Sensitivity, listed at 6.5mV/g with −3 dB bandwidth at 1.6 kHz, is sufficiently high for theapplication. When a post-impact player is removed from a game and notallowed to return until cleared by a concussion-savvy healthcareprofessional, most will recover quickly. If the injury is undetected,however, and an athlete continues playing, concussion recovery oftentakes much longer. In addition, the industry is finding that long-termproblems from delayed or unidentified injury can include: Earlydementia, Depression, Rapid brain aging, and Death. The cumulativeeffects of repetitive head impacts (RHI) increases the risk of long-termneuro-degenerative diseases, such as Parkinson's disease, Alzheimer's,Mild Cognitive Impairment, and ALS or Lou Gehrig's disease. The sensors'most important role is to alert to dangerous concussions. Yet, the actof real-time monitoring brings these players to the attention of theircoaches not only to monitor serious impacts but, based on the dataprovided by the sensors, also help to modify a player's technique sothat they are not, for example, keeping their head low where they cansustain injury to the front and top of the skull. In the NFL there alsohas been an aggressive crackdown against hits to the head and neck—aresponse to the ongoing concussion crisis-resulting in immediate penaltyto players using their helmets as a “weapon”. Customized mouthguardsalso have sensors therein. A customized mouthguard has tested to be 99percent accurate in predicting serious brain injury afternear-concussive force, according to an Academy of General Dentistrystudy2. Teeth absorb and scatter infrared light, which shows how muchforce is taking place at the moment of impact.

Features of the smart protective gear can include:

1. A smart protection gear, comprising:

a wearable surface;

a processor in the band and coupled to a wireless transceiver;

a camera coupled to the surface;

a sensor disposed in the surface; and

an accelerometer disposed within the band to detect acceleration of thesurface.

2. The protection gear of claim 1, comprising a plurality of smartprotection gears forming a mesh network and communicating episodicallyto conserve power.

3. The protection gear of claim 1 where the electronic components,sensors, and interconnects of the protection gear monitor, record,process and/or transmit events of interest (such as accelerometers andgyroscopes for impact events, temperature sensors for temperature and/ortemperature gradients, pressure sensors, moisture sensors, chemicalsensors).

4. The protection gear of claim 1 comprised for sensing and/ormonitoring impact events where the sensors are accelerometers,gyroscopes, and/or pressure sensors.

5. The protection gear of claim 1 comprised for sensing and/ormonitoring and/or controlling ongoing events where the sensors monitortemperature, temperature gradients, motion, position, environmental orchemical levels, or other such information.

6. The protection gear of claim 1 comprised for sensing events or otherinformation including mounting multiple distributed sensors forobtaining spatial and/or temporal distribution in the data and/ormultiple sensors sensing different information and data.

7. The protection gear of claim 1 including wired or wirelesscommunication, such as a Bluetooth module or a wi-fi module or othertransmission module, transmitting and/or receiving information to/fromanother device.

8. The protection gear of claim 1 with power and energy sourcesincluding batteries, wired or wireless rechargeable batteries,photovoltaics, thermoelectrics, or energy harvesters.

9. The protection gear of claim 1 with an adhesive backing for directlyadhering to a head, a body, or an object.

10. The protection gear of claim 1 contained in an adhesive or a sleevefor adhering or attaching to a head, a body, or an object.

11. The protection gear of claim 1 coated with a coating for protectionagainst the elements (water, moisture, dirt, other) and/or for increasedcomfort to the wearer.

12. The protection gear of claim 1, comprising a geometrically patternedsubstrate that contains regions of low strain domains (LSDs) bridged byhingeable strain relief features (SRFs) which also contain low strainregions and enable the stretching, flexing and twisting of theprotection gear while maintaining continuous low strain regions formounting electronic components and routing traces.

13. The protection gear of claim 1 for attachment to or on or an object,or embedded in an object.

14. The protection gear of claim 1 in the form factor of a rectangularor a square or a triangular or other polygon or circular or ellipticalor other geometric shape bandage.

15. The protection gear of claim 1 in the form factor that is orcontains any combination of rectangles, triangles, circles, ellipses orother form factors.

16. The protection gear of claim 1 with different geometric patterningof different numbers and shapes and orientations of low strain domains,different numbers and orientation of geometrically structured hinge-likedomains, and different geometries of hinge-like domains.

17. The protection gear of claim 1 as a programmable circuit board forarbitrary applications.

18. The protection gear of claim 1 fabricated using current flex circuitmanufacturing methods and materials.

19. The protection gear of claim 1 comprising a cloud storage to receivesensor data.

20. The protection gear of claim 1 where the polymer layers are currentflex manufacturing polymers such as Kapton, polyimides, polyamides,polyesters, or other as well as elastomers such as silicone rubbers(PDMS) or polyurethanes or other elastomers and the interconnects aremetals that have high electrical conductivity, such as copper or gold,or where the interconnects are conductive inks.

Custom Gear

In one aspect, the protective gear is custom formed to the athlete'sbody. This is done in FIG. 12C as follows:

-   -   321) perform 3D scan of person and create 3D model    -   322) form positive mold from the 3D model    -   323) place mold into 2 phase 3D printer to form a negative    -   324) put composite material into mold and form composite        protection gear    -   325) embed IoT electronics into one or more locations into the        composite protection gear    -   326) link IoT electronics with mobile devices and cloud based        storage and process impact data and warn user if impact is        unsafe.

The protection gear or footwear can be custom produced at the request ofa customer, who can specify the nature of the customization for one ormore pairs of helmet, protective gear, or footwear. Each helmet of thefootwear may have a different design, message or message portiondesigned into it and rendered using the bed of pins described below tomake the custom helmet or shoe design messages or shapes, and then thebottom sole can be fabricated using the reformable bed described below.Once the negative is fixed in the reformable bed, suitable materials forthe bottom sole can be deposited and cured and can include rubber,plastic, or foam. Further customization can be done by a ComputerizedNumerical Control (CNC) where component design can be integrated withcomputer-aided design (CAD) and computer-aided manufacturing (CAM)programs. The device can be programmed to use a number of differenttools-drills, saws, and so on. Alternatively a number of differentmachines can be used with an external controller and human or roboticoperators that move the component from machine to machine. Regardless, aseries of steps needed to produce a part can produce a part that closelymatches the original CAD design in a highly automated fashion. Inaccordance with aspects of the subject matter disclosed herein throughthe use of reformable bed and a suitably programmed CNC tools,customized footwear with custom cut sole designs, can cost effectivelybe created in small quantities and yet scalable for mass-customization.

1. A method of producing a component of customized wearable protectiongear, the method comprising:

capturing the 3D model of a person and adjusting the 3D model tocustomize a shape to optimize protection or performance;

using a rapid prototyping machine such as 3D printer or a bed of pins torender a positive model of the shape; and

-   -   impressing the positive model into a reformable mold to form the        component of the wearable protective gear.

2. The method of claim 1, wherein the component comprises a helmet,protective padding, shoulder padding, seat, shoe, or sole.

3. The method of claim 1, comprising fabricating a plurality ofcomponents in parallel.

4. The method of claim 1, wherein the component comprises shin guard,Helmet, Chin Strap, Chin Shields, Cup, Athletic Supporter, Elbow Sleeve,Arm Pad, Back Plate, Rib Protection, Facemask, Girdle, Helmet Visor,Shoulder Pad, Hip & Tail Pad, Mouthguard, Neck Roll, Knee Pad, Boot.

5. The method of claim 1, comprising joining the component with an upperto form a shoe.

6. The method of claim 5, wherein the shoe comprises a jogging shoe,basketball shoe, soccer shoe, running shoe, climbing shoe, flip flop,sandal, or boot.

7. The method of claim 1, wherein the reformable mold comprises sandhaving a liquid state and a solid state.

Shock Protection

In one embodiment, the sole is not completely filled with material, butis formed as a lattice structure. The system generates triangulatedsurfaces for export to additive manufacturing (AM) processes.Implementing a process that coverts a CAD object into an image, known asvoxelisation, the company uses an image-based method which allowsdesigners to generate implicitly defined periodic lattice structuressuitable for additive manufacturing applications and finite elementanalysis (FEA). The system generates robust lattice structures canovercome the problems faced with hollowing out a part to reduce weightand optimize designs prior to 3D printing. Cellular lattice structurescan be used to replace the volume of CAD and image-based parts, reducingweight whilst maintaining optimal performance. In this way, the shoescan be light weight yet strong and provide shock impact absorptionduring miming for the wearer.

Topology optimization can be used to drive the material layout includingthe lattice regions. From this new topology optimization implementation,the system can identify void regions in the design space, where thematerial can be removed, regions where solid material is needed, andregions where lattice structure is required. This allows the system togenerate the optimal hybrid or blended solid-lattice design based ondesired functionality of the part.

Lattice structures can be considered as porous structures. In the caseof topology optimization, the semi-dense elements are like the porousmedia. To refine the design, a second-phase involves a detailed sizingoptimization where the end diameters of each lattice cell member areoptimized. This allows for further weight reduction while meeting designrequirements, such as buckling, stress, and displacement.

A piezo material can be actuated to generate a vibration that cancelsincoming shock on the wearer. In one embodiment, the system tracks theshock such as the foot contact patterns and generates an anti-vibrationsignal to cancel the shock generated when the foot contacts the ground.In this embodiment, a processor receives foot ground contact using anaccelerometer. The stride pattern is determined, and the next footground contact is detected, and the piezo material is actuated with acounter signal to cancel the expected shock. This is similar to thenoise cancellation, except the vibration/shock is canceled.

In one hybrid embodiment, the shoes incorporate passive and activeisolation elements. The passive component consists of springs whichsupport the load weight and provide isolation over a broad spectrum.These springs provide a basic level of isolation in the lowerfrequencies and excellent isolation in the higher frequencies (above 200Hz). They also support the load while allowing for travel of theactuators in the active component. The performance of the springs isaugmented and corrected by an active isolation component. The activeisolation component consists of vibration sensors, control electronics,and actuators. The vibration sensors are piezo accelerometers. Aplurality of sensors in each isolation system are positioned indifferent orientations to sense in all six degrees of freedom. The piezoaccelerometers convert kinetic vibration energy into electrical signalswhich are transmitted to the control electronics. The electronicsreconcile and process the signals from the various sensors using aprocessor. The electronics then send a cancellation signal to theactuators. The actuators generate vibrations that are equal to theincoming vibrations but out of phase in relation to the incomingvibrations. This results in cancellation of the incoming vibrationalnoise, leaving the wearer undisturbed. This process occurs within 5-20milliseconds of a vibration entering the system.

Car Repair/Maintenance

The sensors can be part of a car, a motorcycle, a bicycle, a boat, aplane, a dirigible, or a drone, for example. FIG. 13 shows an exemplarysmart vehicle. The sensors can be used for maintenance prediction and incase of component failure, to help the driver to navigate safely. Forexample, the vehicle can monitor brake pad wear and adjusting how hardthe brake needs to be applied in light of other vehicles and how fastdoes the vehicle need to come to a complete stop. In addition tochanging the way the vehicle brake, the vehicle may change the way itmaneuvers in other ways as well, such as accelerating differently orchanging directions. For instance, the vehicle may accelerate moreslowly if the measured oil pressure is excessively high. The vehicle mayalso turn more or less tightly in order to mitigate wear. The vehiclemay also use other systems and methods to determine the state of avehicle component. For example, the vehicle may monitor how far it takesthe car to stop compared to expected braking distance. If the distanceis longer than expected, such as taking longer than it has in the past,the computer system may determine that the brakes are worn and startbraking earlier. The system and method may also estimate the state of acomponent based on its repair service record. In that regard, theprocessor may query data 134 or an external database (e.g., a serverwith which the vehicle is in wireless communication) for repair recordsand estimate the wear on a component based on the length of time sincethe last repair.

The system and method may rely on other information to change the waythe vehicle is maneuvered. For instance, the vehicle may sense weightdistribution and adjust maneuvering in response to the changes in theloading and/or weight distributions on the vehicle. The vehicle mayfurther move differently when there is only one user in the vehicle thanfour passengers on board, or differently with light loads than withhauling a trailer behind. The vehicle may also adapt the driving to theobserved environmental changes such as weather or roadway conditions.

Modeling of the patterns of changes in the vehicle's performance andconditions, as well as modeling of the patterns of changes in thedriving environment, may be performed by the autonomous driving computersystem. Alternatively, predetermined models may be stored in theautonomous driving system. The computer system may process the observeddata, fit them into the 3D models in FIGS. 7A-7I, and issue compensationsignals accordingly.

The vehicle may take the steps necessary to repair a component. By wayof example, when the vehicle is not being used by anyone, the vehiclemay autonomously and without direct human assistance navigate to arepair facility, notify the facility of the component that requiresrepair and return to its original location when the repair is finished.

Gesture Sensor for Vehicular Control

In an exemplary gesture recognition system. The system takes advantageof the numerous cameras onboard the vehicle for navigation and mappingpurposes, and additionally includes the gesture control feature. Thecameras can be any type of camera, including cameras sensitive acrossthe visible spectrum or, more typically, with enhanced sensitivity to aconfined wavelength band (e.g., the infrared (IR) or ultraviolet bands);more generally, the term “camera” herein refers to any device (orcombination of devices) capable of capturing an image of an object andrepresenting that image in the form of digital data. For example, linesensors or line cameras rather than conventional devices that capture atwo-dimensional (2D) image can be employed. The term “light” is usedgenerally to connote any electromagnetic radiation, which may or may notbe within the visible spectrum, and may be broadband (e.g., white light)or narrowband (e.g., a single wavelength or narrow band of wavelengths).The cameras are preferably capable of capturing video images (i.e.,successive image frames at a constant rate of at least 15 frames persecond), although no particular frame rate is required. The capabilitiesof cameras are not critical to the invention, and the cameras can varyas to frame rate, image resolution (e.g., pixels per image), color orintensity resolution (e.g., number of bits of intensity data per pixel),focal length of lenses, depth of field, etc. In general, for aparticular application, any cameras capable of focusing on objectswithin a spatial volume of interest can be used. For instance, tocapture motion of the hand of an otherwise stationary person, the volumeof interest might be defined as a cube approximately one meter on aside. For example, lasers or other light sources can be used instead ofLEDs. For laser setups, additional optics (e.g., a lens or diffuser) maybe employed to widen the laser beam (and make its field of view similarto that of the cameras). Useful arrangements can also include short- andwide-angle illuminators for different ranges. Light sources aretypically diffuse rather than specular point sources; for example.

In identifying the location of an object in an image according to anembodiment of the present invention, light sources are turned on. One ormore images are captured using cameras. In some embodiments, one imagefrom each camera is captured. In other embodiments, a sequence of imagesis captured from each camera. The images from the two cameras can beclosely correlated in time (e.g., simultaneous to within a fewmilliseconds) so that correlated images from the two cameras can be usedto determine the 3D location of the object. A threshold pixel brightnessis applied to distinguish object pixels from background pixels. This canalso include identifying locations of edges of the object based ontransition points between background and object pixels. In someembodiments, each pixel is first classified as either object orbackground based on whether it exceeds the threshold brightness cutoff.Once the pixels are classified, edges can be detected by findinglocations where background pixels are adjacent to object pixels. In someembodiments, to avoid noise artifacts, the regions of background andobject pixels on either side of the edge may be required to have acertain minimum size (e.g., 2, 4 or 8 pixels).

In other embodiments, edges can be detected without first classifyingpixels as object or background. For example, Δβ can be defined as thedifference in brightness between adjacent pixels, and |Δβ| above athreshold can indicate a transition from background to object or fromobject to background between adjacent pixels. (The sign of Δβ canindicate the direction of the transition.) In some instances where theobject's edge is actually in the middle of a pixel, there may be a pixelwith an intermediate value at the boundary. This can be detected, e.g.,by computing two brightness values for a pixel i: βL=(βi+βi−1)/2 andβR=(βi+βi+1)/2, where pixel (i−1) is to the left of pixel i and pixel(i+1) is to the right of pixel i. If pixel i is not near an edge,|βL−βR| will generally be close to zero; if pixel is near an edge, then|βL−βR| will be closer to 1, and a threshold on |βL−βR| can be used todetect edges.

In some instances, one part of an object may partially occlude anotherin an image; for example, in the case of a hand, a finger may partlyocclude the palm or another finger Occlusion edges that occur where onepart of the object partially occludes another can also be detected basedon smaller but distinct changes in brightness once background pixelshave been eliminated.

Detected edges can be used for numerous purposes. For example, aspreviously noted, the edges of the object as viewed by the two camerascan be used to determine an approximate location of the object in 3Dspace. The position of the object in a 2D plane transverse to theoptical axis of the camera can be determined from a single image, andthe offset (parallax) between the position of the object intime-correlated images from two different cameras can be used todetermine the distance to the object if the spacing between the camerasis known.

Further, the position and shape of the object can be determined based onthe locations of its edges in time-correlated images from two differentcameras, and motion (including articulation) of the object can bedetermined from analysis of successive pairs of images. An object'smotion and/or position is reconstructed using small amounts ofinformation. For example, an outline of an object's shape, orsilhouette, as seen from a particular vantage point can be used todefine tangent lines to the object from that vantage point in variousplanes, referred to herein as “slices.” Using as few as two differentvantage points, four (or more) tangent lines from the vantage points tothe object can be obtained in a given slice. From these four (or more)tangent lines, it is possible to determine the position of the object inthe slice and to approximate its cross-section in the slice, e.g., usingone or more ellipses or other simple closed curves. As another example,locations of points on an object's surface in a particular slice can bedetermined directly (e.g., using a time-of-flight camera), and theposition and shape of a cross-section of the object in the slice can beapproximated by fitting an ellipse or other simple closed curve to thepoints. Positions and cross-sections determined for different slices canbe correlated to construct a 3D model of the object, including itsposition and shape. A succession of images can be analyzed using thesame technique to model motion of the object. Motion of a complex objectthat has multiple separately articulating members (e.g., a human hand)can be modeled using these techniques.

More particularly, an ellipse in the xy plane can be characterized byfive parameters: the x and y coordinates of the center (xC, yC), thesemimajor axis, the semiminor axis, and a rotation angle (e.g., angle ofthe semimajor axis relative to the x axis). With only four tangents, theellipse is underdetermined. However, an efficient process for estimatingthe ellipse in spite of this fact involves making an initial workingassumption (or “guess”) as to one of the parameters and revisiting theassumption as additional information is gathered during the analysis.This additional information can include, for example, physicalconstraints based on properties of the cameras and/or the object. Insome circumstances, more than four tangents to an object may beavailable for some or all of the slices, e.g., because more than twovantage points are available. An elliptical cross-section can still bedetermined, and the process in some instances is somewhat simplified asthere is no need to assume a parameter value. In some instances, theadditional tangents may create additional complexity. In somecircumstances, fewer than four tangents to an object may be availablefor some or all of the slices, e.g., because an edge of the object isout of range of the field of view of one camera or because an edge wasnot detected. A slice with three tangents can be analyzed. For example,using two parameters from an ellipse fit to an adjacent slice (e.g., aslice that had at least four tangents), the system of equations for theellipse and three tangents is sufficiently determined that it can besolved. As another option, a circle can be fit to the three tangents;defining a circle in a plane requires only three parameters (the centercoordinates and the radius), so three tangents suffice to fit a circle.Slices with fewer than three tangents can be discarded or combined withadjacent slices.

To determine geometrically whether an object corresponds to an object ofinterest comprises, one approach is to look for continuous volumes ofellipses that define an object and discard object segments geometricallyinconsistent with the ellipse-based definition of the object—e.g.,segments that are too cylindrical or too straight or too thin or toosmall or too far away—and discarding these. If a sufficient number ofellipses remain to characterize the object and it conforms to the objectof interest, it is so identified, and may be tracked from frame toframe.

In some embodiments, each of a number of slices is analyzed separatelyto determine the size and location of an elliptical cross-section of theobject in that slice. This provides an initial 3D model (specifically, astack of elliptical cross-sections), which can be refined by correlatingthe cross-sections across different slices. For example, it is expectedthat an object's surface will have continuity, and discontinuousellipses can accordingly be discounted. Further refinement can beobtained by correlating the 3D model with itself across time, e.g.,based on expectations related to continuity in motion and deformation.In some embodiments, light sources 808, 110 can be operated in a pulsedmode rather than being continually on. This can be useful, e.g., iflight sources 808, 110 have the ability to produce brighter light in apulse than in a steady-state operation. Light sources 808, 110 can bepulsed on at regular intervals as shown at 502. The shutters of cameras802, 804 can be opened to capture images at times coincident with thelight pulses. Thus, an object of interest can be brightly illuminatedduring the times when images are being captured. In some embodiments,the silhouettes of an object are extracted from one or more images ofthe object that reveal information about the object as seen fromdifferent vantage points. While silhouettes can be obtained using anumber of different techniques, in some embodiments, the silhouettes areobtained by using cameras to capture images of the object and analyzingthe images to detect object edges.

Hand-Gesture Control of Vehicle

Exemplary hand control of a smart vehicle. First, a Left Hand GestureBased Car Control process is disclosed. In an exemplary left arm gesturebased window glass control process, the process checks for the raisedarm (1002). If the arm is raised (1004) it checks for the number offingers raised (1006). The controls for windows are activated if firstfour fingers are raised (1008). The process allows controlling only thedriver seat glass control and also all the window glass control. Thisdecision is based on the number of fingers raised. A single fingerchooses only driver glass. Movements of the glass is than controlled bythe angular movement of the arm (1020), a right movement slides theglass up (1028) and a left movement slides it down (1026). The processis concluded (1030) after the windows are at the required position. Atany moment the driver can choose to exit the process by forming a firstof his left arm.

A seat control process is capable of controlling both the driver seatand the front passenger seat as well. The process starts with checkingfor which arm is raised (1036). After the arm the process scans for thefingers (1038), first 2 fingers initiate the seat actuators (1040). Now,the driver can choose to adjust his own seat or maybe the seat of thepassenger. This decision is dependent whether one or two fingers areraised (1044). The seats can be moved forth or back as per the armsangular movement (1052) (1050). As per the convenience, the seats can beadjusted. After the adjustment is done, the process concludes (1062). Atany point the process can be ended if the driver forms a first on lefthand.

A left hand based gesture based mechanism for unlocking the ‘Hood’ andthe ‘Trunk’ of the car is next. As it is left arm based control, themechanism uses a camera to check the arm raised. A raised left arminitiates the process which unlocks the hood and trunk (1068). Thecamera than checks for the fingers that are raised (1070), the firstfinger is used to activate the hood & trunk control (1072). To open thetrunk the driver has to make a right angular movement (1076) and anopposite movement for unlocking the hood (1078). As soon as either ofthe two is unlocked the process ends (1082). If the process is startedby mistake or confusion, the driver can choose to exit by forming afirst on his left arm.

An exemplary process for controlling temperature of the driver and frontpassenger seats is next. After checking for left raised arm (1088) thecamera scans for the fingers raised (1090). The first three fingers areto be used by the driver to activate seat temperature controls (1092).The driver can choose to control his seat temperature or the passenger'sseat temperature by raising the appropriate number of fingers (1096).The angular movements of the left arm can be used to increase ordecrease the temperature of the selected seat (1104) (1102). The processcan be ended after adjusting the temperature or at any other point byforming a first (1114).

An left arm gesture based navigation (GPS) control for a car is next.The process initializes when the driver raises his/her left arm (1120).The GPS system is activated if the all the fingers are raised i.e. anopen palm (1124). Now the arm motion in the vertical and horizontal axiscan be used to move the GPS pointer (1128). To select a particulardestination, the pointer must be kept at the same location for apre-defined duration of time (1144). Once the destination is set, theGPS starts routing (1146) and then exits the process (1148). The processcan be ended abruptly if needed by forming a first on left hand.

An exemplary gesture based control of drivers mirror using left arm isnext. The driver initiates the process by raising the left arm (1154).The thumb is used as a trigger for activating the mirror actuators(1158). To adjust the mirror angle, the driver can move his/her armalong the vertical or horizontal axis (1162). The driver can form afirst (1170) (1168) or wait for a predefined time interval to set themirror angle (1182). This process has an option which enables the driverto exit anytime by forming a first on left hand.

An exemplary music control in the car using gestures of right hand isnext. The process is activated if the camera scans a vertically standingright arm (1190). The car music system is initiated if the driver has anopen right palm (1194). Depending upon the fingers raised after themusic system is initiated either radio or just the MP3 player is started(1204). The angular movements of the arm can be used to switch betweenstations or songs (1206) (1202). Once the desired station or song isselected the driver can exit the process by forming a closed first(1216). A closed first formed anytime can be used to exit the processanytime.

An exemplary car temperature control using gestures from the right armfollows. The driver is expected to raise the first two fingers of theright arm to activate the temperature controls (1246). The temperaturecontrolling element is the angular motion of the right arm (1250). Aleft motion causes decrease in temperature and vice versa (1256) (1254).Once the desire temperature is achieved, the driver can stop the processby forming a fist. A first basically exits the process at any givenpoint.

An exemplary control the car volume using arm gestures is next. Thecamera initiates the process whenever the driver raises his/her rightarm (1222). The process expects the driver to raise three fingers toinitiate volume control (1226). Using the right or left angular motionthe volume can be increased and decreased (1230).

An exemplary technique for sliding the sun roof by the means of handgesture can be as follows. The sun roof control process starts when thedriver raises his/her right arm (1264) and first four fingers of thesame (1268). The camera now scans for the angular motion of the arm(1272). A left motion pulls the roof back (1276) whereas a right motionpushes it forward so that it can be closed (1274). The process ends oncethe roof is entirely opened or closed and it can also be concluded byforming a first on the right arm.

An exemplary arm gesture based technique for controlling the car windshield wipers is next. The wiper motors are activated when the right armalong with first finger is raised (1284) (1288). The speed of the wipermotors can be controlled using the right arm angular motion (1292). Theleft motion decreases the speed (1298), the right motion increases thewiper speed (1296) and in order to stop the wiper a still right arm witha closed first should be scanned by the camera (1294).

An exemplary right arm gesture based control of the rear view mirror isnext. The camera scans for the right arm, if it is up the process isinitiated (1306). The rear view mirror control is activated if thecamera scans only a thumb on right arm (1310). Now, the rear view mirrorcan be adjusted vertically and horizontally, this is achieved by movingthe arm with only raised thumb along the desired axis (1314). To lockthe position of the mirror, the same position is to be maintained for apre-defined interval of time. Once done the process locks the mirror andconcludes (1332). The process can be ended anytime by the driver byforming a first on his right arm.

In other embodiments, by cupping the hand on an object such as asteering wheel, the user can use voice to make calls, receive andrespond to texts, launch apps, get turn-by-turn directions, find thenearest Chinese restaurant and other local businesses, or say “Play mesome Barry Manilow.” You can also ask Siri or Google Now to search theInternet as you roll down the Interstate. The apps will be able to pullcontacts directly from the phone's address book, access favorites andbookmarks, and have user location history close at hand.

A gesture is used to control an air conditioning system in an examplevehicle, in accordance with an embodiment. The vehicle may maintain acorrelation between a plurality of predetermined gestures, incombination with a plurality of predetermined regions of the vehicle,and a plurality of functions, such that each gesture in the plurality ofpredetermined gestures, in combination with a particular region of theplurality of predetermined regions, is associated with a particularfunction in the plurality of functions, as described above. For example,the correlation may include a downward swiping gesture in a region thatincludes an air-conditioning vent associated with the function ofdecreasing a fan speed of an air conditioning system. Other examples arepossible as well.

As shown, a fan speed indicator on the display indicates that a fanspeed of the air conditioning system in the vehicle is high. At somepoint, the user may wish to lower the fan speed of the air conditioningsystem. To this end, the user may make a downward swiping gesture in aregion that includes an air-conditioning vent. The camera 304 may recordthree-dimensional images of the downward swiping gesture in the regionthat includes an air-conditioning vent. Based on the three-dimensionalimages, the vehicle may detect the downward swiping gesture in theregion that includes the air-conditioning vent.

The vehicle may then select, based on the correlation, a functionassociated with the downward swiping gesture in the region that includesthe air-conditioning vent. For example, the downward swiping gesture inthe region that includes the air-conditioning vent may be associatedwith the function of decreasing a fan speed of the air conditioningsystem, as described above. Other examples are possible as well. Oncethe vehicle has selected the function from the correlation, the vehiclemay initiate the function in the vehicle. That is, the vehicle maydecrease the fan speed in the vehicle.

In some embodiments, the vehicle may additionally determine an extentdetermining an extent of the downward swiping gesture and may decreasethe fan speed by an amount that is, for example, proportional to theextent.

In some embodiments, in addition to initiating the function, the vehiclemay trigger a feedback to the user, such as an audible feedback, avisual feedback, and/or a haptic feedback. Such feedback may beparticularly useful when the function is not immediately detectable bythe user, such as a small decrease in the fan speed of the climatecontrol system or a slight repositioning of a seat.

Further, in some embodiments, the vehicle may determine an extent of thegiven gesture. For example, if the given gesture is a swipe gesture, thevehicle may determine an extent of the swipe (e.g., how long the swipeis in space and/or time). The vehicle may then determine an operationalparameter based on the extent. For example, for a greater extent, thevehicle may determine a greater operational parameter than for a lesserextent. The operational parameter may be, for example, proportional to,or approximately proportional to, the extent. In these embodiments, whenthe vehicle initiates the function the vehicle may initiate the functionwith the determined operational parameter.

For example, if the swipe gesture is in a region that includes a window,and the swipe gesture in the region that includes the window isassociated with opening the window, the vehicle may determine an extentof the swipe and further may determine how far to open the window basedon the extent of the swipe. For instance, the vehicle may open thewindow further for a longer swipe than for a shorter swipe.

As another example, if the swipe gesture is in a region that includes anair-conditioning vent, and the swipe gesture in the region that includesthe air-conditioning vent is associated with lowering a temperature inthe vehicle, the vehicle may determine an extent of the swipe andfurther may determine how much to lower the temperature in the vehiclebased on the extent of the swipe. For instance, the vehicle may lowerthe temperature further for a longer swipe than for a shorter swipe.

Such an extent could be determined for gestures other than a swipegesture as well. For example, if a tap gesture is in a region thatincludes a speaker, and the tap gesture in the region that includes thespeaker is associated with lowering a volume of an audio system, thevehicle may determine an extent of the tap (e.g., how many taps, howlong the tap is held, etc.) and further may determine how much to lowerthe volume of the audio system based on the extent of the tap. Forinstance, the vehicle may lower the volume more for more taps (or alonger tap) than for fewer taps (or a shorter tap).

In some embodiments, rather than determining the extent of the gestureand the corresponding operational parameter and then initiating thefunction with the determined operational parameter, the vehicle mayinstead continuously determine the extent of the gesture and update thecorresponding operational parameter, and may continuously initiate thefunction with the updated operational parameter. For example, thevehicle may detect a cover gesture in a region that includes anair-conditioning vent (e.g., such that the air-conditioning vent iscovered), and the cover gesture in the region that includes theair-conditioning vent may be associated with lowering a fan speed of theair conditioning system. Once the vehicle detects the cover gesture inthe region that includes the air-conditioning vent, the vehicle maylower the fan speed (e.g., by a predetermined amount). As the vehiclecontinues to detect the cover gesture, the vehicle may continue to lowerthe fan speed (e.g., in increments of, for example, the predeterminedamount, growing amounts, etc.). Once the vehicle detects that the covergesture has ended, the vehicle may cease to lower the fan speed. As aresult, during the cover gesture the vehicle may lower the fan speed byan amount that is based on the extent of the cover gesture.

In some embodiments, the vehicle may have difficulty detecting the givengesture and/or the given region. For example, the vehicle may determinethat a confidence level of one or both of the given gesture and thegiven region is below a predetermined threshold. In these embodiments,the vehicle may request an occupant to repeat the given gesture in thegiven region. When the occupant repeats the given gesture in the givenregion, the vehicle may record additional three-dimensional images andmay detect the given gesture and the given region based on theadditional three-dimensional images (and, in some cases, thethree-dimensional images previously recorded).

Obstacle Detection

In some embodiments, a vehicle identifies obstacles on the road, and thecomputer system may use one or more sensors to sense the obstacles. Forexample, the computer system may use an image-capture device to captureimages of the road and may detect the obstacles by analyzing the imagesfor predetermined colors, shapes, and/or brightness levels indicative ofan obstacle. As another example, the computer system may project LIDARto detect the obstacle. The computer system may estimate the location ofthe obstacle and control the vehicle to avoid the vehicle and yetmaintain a predetermined distance from neighboring vehicles in bothdirections. Other vehicles behind the lead vehicle can then simplyfollow the lead vehicle as part of a flock. The computer system may thencontrol the vehicle to maintain a distance between the vehicle and theat least one neighboring vehicle to be at least a predetermined minimumdistance to avoid colliding with the at least one neighboring vehicle.

In an example of a situation where an obstacle 603 is present in frontof a host vehicle 601 mounted with a sensor such as camera or sensor 602in a front portion of a vehicle body. In vehicles, a vehicle-mountedsensors such as cameras, radar, and LIDAR is used in a vehicle controlsystem such as an inter-vehicle distance alarm system, a precedingvehicle following system or a collision reducing brake system. Where anobstacle is not present in front of the host vehicle 601, since thetarget data is not output from the sensor 602, the vehicle velocitycontrol system performs a control so that the vehicle operates accordingto a planned path. However, the path may need adjustment when anobstacle is encountered, or when weather affects the operation, ortraffic condition, emergency or holiday patterns require a change in theplanned path and speed.

In this example, the front obstacle 603 is another vehicle. Furthermore,in this example, as the sensor 602, a radar in one embodiment, hashorizontal resolution due to a plurality of arrays installed in thehorizontal direction; however, it does not have a vertical resolution.In this case, the sensor 602 outputs target data having positioninformation such as a relative longitudinal distance, lateral positionand velocity between the host vehicle 601 and the obstacle 603 to avehicle velocity control system. In another embodiment, the sensor 602is a camera. Pictures captured by the camera can be used to form a 3Dreconstruction of the obstacles and the road. The task of convertingmultiple 2D images into 3D model consists of a series of processingsteps: Camera calibration consists of intrinsic and extrinsicparameters, and the camera calibration is usually required fordetermining depth. Depth determination calculates—depth. Thecorrespondence problem, finding matches between two images so theposition of the matched elements can then be triangulated in 3D space.With the multiple depth maps the system combines them to create a finalmesh by calculating depth and projecting out of the camera—registration.Camera calibration will be used to identify where the many meshescreated by depth maps can be combined together to develop a larger one,providing more than one view for observation to have a complete 3D mesh.

The vehicle velocity control system performs controls such as a controlfor maintaining the distance from the obstacle 603, and a control forexecuting an alarm or velocity reduction in a case where collision withthe obstacle 603 is predicted, according to the position informationabout the input target data.

For example, obstacles such as land slip and falling rocks can appearunexpectedly on a mountain road. Image data in the fall monitoring areais processed by car computers and/or transmitted to a cloud processingsystem, and images are frame-difference-processed to identify the newobstacles. Thereafter, in the case where there are no variations in theextracted component data representing the body and in particular in thecase where there are no moving variations, when several frames with aninterval of a predetermined time are processed in a similar way, theprocessor detects the obstacle and transmits a warning signal toadjacent vehicles and/or to a traffic display board. The warninginformation can be incorporated by the driving software of other driversthat there are road obstacles by land slip.

In another embodiment, the obstacle can be the result of a car accidentor emergency. The system automatically detects the occurrence of anemergency and provides safety at the scene. This is done by divertingtraffic flow near the point of emergency to a point where trafficresumes normal flow. The system secures the incident site to protectemergency personnel, their equipment and the public, from hazardousconditions at the scene and throughout the traffic control zone. Thesystem can establish a traffic control set-up that gives motoristsadequate warning and reaction time. The system also separatespedestrians from vehicular traffic and limits access to the site toauthorized persons only. One embodiment directs vehicles through anemergency traffic control zone with the following: Advance Warning Areashould alert vehicles that there is a traffic situation or difficultyahead which will require some action on its part; Approach area shouldidentify the nature of the equipment or vehicle that is about toencounter and allow them to analyze the situation; Transition Areashould provide an indication as to the expected action to be taken bythe vehicle to decide on a course of action and execute safe drivingtechniques prior to entering the Activity Area; and Activity Areaincludes Fend Off Position of the emergency vehicle, Buffer Zone (refersto scene protection area between the first emergency vehicle and theincident site), Incident Site (Restricted to authorized personnel only),Traffic Space (Area where traffic is allowed to pass by the ActivityArea), and Staging Area (Emergency Vehicles not immediately required toperform a function or shielding at the incident scene should be directedto stage in this area. The area should be downstream/upstream of theincident site and the location should not create a traffic hazard orobstruction). The system can determine a Termination Area from thedownstream side of the Staging Area to the point where normal traffic isable to resume. The information for an emergency is incorporated intothe 3D model for vehicular processing.

Weather conditions can affect the driving plan. For cameras, it affectsthe ability to see, which is very limited in adverse weather conditionssuch as rain, fog, ice, snow, and dust. For example, if the fog becomesso thick the system can suggest the car be moved completely off theroad. The car system also slows down for rain, drizzle, or snow on theroad. This is when many road surfaces are most slippery because moisturemixes with oil and dust that has not been washed away. The slipperyroads can reduce traction and control of the vehicle may be compromised.The system can detect wet road surface via its camera or water sensors.Wet road surfaces can cause tires to hydroplane (skim on a thin layer ofwater). This could result in loss of control and steering ability.Hydroplaning is caused by a combination of standing water on the road,car speed, and under-inflated or worn-out tires. Thus, the system cancheck the pressure of the tires by communicating with the tire sensors.The 3D modeling system also incorporates the effects of hightemperatures, sun glare and high winds. One exemplary 3D modelingprocess for navigation is detailed next.

An exemplary system can perform data fusion based on sensor baseddetection of objects, change in weather and traffic, andholiday/emergency conditions, among others. The process checks all thesensors for change in weather (2004), detection of object (2002) and theGPS for current traffic conditions (2006). For each given sensor fordetecting objects in a vehicle's environment, the process generates a 3Dmodel of the given sensor's field of view; obstacle information fromfront cars using vehicle-vehicle communication (DRSC); neighboring cardriver preference information; traffic information including emergencyinformation. The process can adjust one or more characteristics of theplurality of 3D models based on the received weather information toaccount for an impact of the actual or expected weather conditions onone or more of the plurality of sensors. After the adjusting,aggregating, by a processor, the plurality of 3D models to generate acomprehensive 3D model; combining the comprehensive 3D model withdetailed map information; and using the combined comprehensive 3D modelwith detailed map information to maneuver the vehicle. In FIG. 7A, theprocess checks sensors for object detection (2008) and then checks forconfirmations from other vehicles over V2V communication such as DSRCand then generates 3D model therefrom. The process can also check forweather change (2004) and correlate the weather change to generate anupdated 3D model. Similarly, the process integrates traffic flowinformation (2006) and updates the 3D model as needed. FIG. 7B shows anexemplary process for identifying the object, while FIG. 7C-7H show inmore details the object modeling process. The process checks sensors forobject detection and scans the object against 3D library for matches. Ifa match is found, the process sets the object to the object in thelibrary, and otherwise the process performs a best-guess of what theobject is and send the object identification for subsequent 3D modelinguse.

Exemplary detection of objects outside of the vehicle and guidance ontheir handling is next. The detected objects can include automobile, apedestrian, structure, or a bicycle, for example. The system assists thedriver by identifying the objects as potential “threats” and recommendoptions for the driver. For example, the system can perform thefollowing:

-   -   detecting an object external to a vehicle using one or more        sensors;    -   determining a classification and a state of the detected object;    -   estimating the destination of the object;    -   predicting a likely behavior of the detected object based on        prior behavior data and destination;    -   preparing the vehicle to respond based at least in part on the        likely behavior of the detected object; and    -   notifying a driver of options based on the likely behavior.

For example, the state of the detected object can be related to at leastone of: location, traffic lane in which the detected object istraveling, speed, acceleration, entry onto a road, exit off of a road,activation of headlights, activation of taillights, or activation ofblinkers. The behavior data is based on movement data for a plurality ofother objects at one or more locations. The movement data are trackedusing one of: satellite imagery, roadside cameras, on-board GPS data, orsensor data acquired for other nearby vehicles. The system can send adriver recommendation or vehicle command to orient the vehicle includespositioning the vehicle at a predetermined distance from the detectedobject, the predetermined distance being based, at least in part, on theclassification of the detected object. The likely behavior of thedetected object can be provided as a probability of the detected objectentering to one or more states. The process includes receiving updatedbehavior data; and wherein predicting the likely behavior of thedetected object is based at least in part on the updated behavior data.The driver can be informed of the options using haptic interface or aheads-up display. The process can also share the likely behavior of theobject to neighboring vehicles using vehicle-to-vehicle communication.

The process may cause the vehicle to take particular actions in responseto the predicted actions of the surrounding objects. For example, ifother car is turning at the next intersection, the process may slow thevehicle down as it approaches the intersection. In this regard, thepredicted behavior of other objects is based not only on the type ofobject and its current trajectory, but also based on some likelihoodthat the object may obey traffic rules or pre-determined behaviors. Inanother example, the process may include a library of rules about whatobjects will do in various situations. For example, a car in a left-mostlane that has a left-turn arrow mounted on the light will very likelyturn left when the arrow turns green. The library may be built manually,or by the vehicle's observation of other vehicles (autonomous or not) onthe roadway. The library may begin as a human built set of rules whichmay be improved by the vehicle's observations. Similarly, the librarymay begin as rules learned from vehicle observation and have humansexamine the rules and improve them manually. This observation andlearning may be accomplished by, for example, tools and techniques ofmachine learning. In addition to processing data provided by the varioussensors, the computer may rely on environmental data that was obtainedat a previous point in time and is expected to persist regardless of thevehicle's presence in the environment. For example, the system can usehighly detailed maps identifying the shape and elevation of roadways,lane lines, intersections, crosswalks, speed limits, traffic signals,buildings, signs, real time traffic information, or other such objectsand information. For example, the map information may include explicitspeed limit information associated with various roadway segments. Thespeed limit data may be entered manually or scanned from previouslytaken images of a speed limit sign using, for example, optical-characterrecognition. The map information may include three-dimensional terrainmaps incorporating one or more of objects listed above. For example, thevehicle may determine that another car is expected to turn based onreal-time data (e.g., using its sensors to determine the current GPSposition of another car) and other data (e.g., comparing the GPSposition with previously-stored lane-specific map data to determinewhether the other car is within a turn lane). These objects may haveparticular behavior patterns that depend on the nature of the object.For example, a bicycle is likely to react differently than a motorcyclein a number of ways. Specifically, a bicycle is more likely to makeerratic movements when compared with a motorcycle, but is much slowerand thus can be handled with ease compared to a speeding motorcycle. Foreach classification, the object data may also contain behaviorinformation that indicates how an object having a particularclassification is likely to behave in a given situation. Vehicle maythen autonomously respond to the object based, in part, on the predictedbehavior.

In an exemplary system for crowd-sourcing navigation data, the systemincludes a crowdsourcing server in communication with a plurality ofvehicles 1 . . . n. The vehicles i performs peer-to-peer discovery andcrowd-sourced navigation. The system receives proximity services for agroup of vehicles traveling a predetermined route using peer-to-peerdiscovery, receives crowdsourcing data from said plurality of vehicles,sharing crowdsourcing data to the group of vehicles (or a subsequentgroup of vehicles) traveling the route of interest. Such information canbe used in providing navigation guidance to the vehicle traveling theroute using the crowdsourced data. In one aspect, the vehicles travelingthe same route can be determined using a vehicle to vehiclecommunication protocol that facilitate identifying peers based uponencoded signals during peer discovery in a peer to peer network. Thesystem can be WiFi or cellular based such as the Proximity Services viaLTE Device Broadcast, among others. In one embodiment, theidentification of peers based upon encoded signals during peer discoveryin a peer to peer network can be done. For example, direct signalingthat partitions a time-frequency resource into a number of segments canbe utilized to communicate an identifier within a peer discoveryinterval; thus, a particular segment selected for transmission cansignal a portion of the identifier, while a remainder can be signaledbased upon tones communicated within the selected segment. Moreover, asubset of symbols within the resource can be reserved (e.g., unused) toenable identifying and/or correcting timing offset. Further, signalingcan be effectuated over a plurality of peer discovery intervals suchthat partial identifiers communicated during each of the peer discoveryintervals can be linked (e.g., based upon overlapping bits and/or bloomfilter information). The method can include transmitting a first partialidentifier during a first peer discovery interval. Also, the method cancomprise transmitting a second partial identifier during a second peerdiscovery interval. Further, the method can include generating bloomfilter information based upon the combination of the first partialidentifier and the second partial identifier. Moreover, the method cancomprise transmitting the bloom filter information to enable a peer tolink the first partial identifier and the second partial identifier.Another embodiment communicates using LTE Direct, a device-to-devicetechnology that enables discovering thousands of devices and theirservices in the proximity of ˜500 m, in a privacy sensitive and batteryefficient way. This allows the discovery to be “Always ON” andautonomous, without drastically affecting the device battery life. LTEDirect uses radio signals—called ‘expressions’—which can be private anddiscreet (targeted securely for certain audiences only) or public(transmitted so that any application can receive them). Publicexpressions are a common language available to any application todiscover each other, and this is the door to consumer utility andadoption. Public expressions exponentially expand the field of value.For example, vehicles that share same driving segments can broadcastexpressions indicating their path(s). The system detects vehicles in thesame segment as part of the proximity services for capturing and sharingcrowd-sourced navigation data. Public expressions combine allapplications—all value—into one single network, thereby expanding theutility of the system. The crowdsourcing data includes vehicleperformance information and GPS locations of a vehicle; and wherein thevehicle data includes odometer information, speedometer information,fuel consumption information, steering information. The data includesinformation relating to closing of a lane using the crowdsourcing data;predicting an avoidance maneuver using the crowdsourcing data;predicting a congestion with respect to a segment of the route of the atleast one vehicle using the crowdsourcing data; and predicting trafficlight patterns using the crowdsourcing data. The system can determinethe presence of obstacles in a road lane by monitoring a pattern ofvehicle avoidance of a particular location of the lane. The obstaclescan be rocks or debris on the lane, closure of a lane, inoperativevehicles on the lane, or vehicles suffering from an accident, amongothers. The vehicular avoidance information can be sent to vehicles thatare planning to use that particular road section to optimize travel, forexample. The system can detect closing of a lane by monitoring changesof vehicle direction at a location on the route of the at least onevehicle; and determining a lane is closed in response to a number ofchanges of vehicle direction being larger than a predetermined thresholdvalue. The system can share prior vehicle's avoidance maneuver bymonitoring change of vehicle direction and distance traveled at a closevicinity of a location on the route of a lead vehicle; and determiningan avoidance maneuver in response to a ratio of change of vehicledirection and distance traveled being less than a predeterminedthreshold value. The system can determine a route based at least in parton an amount of time predicted for travelling from a starting locationto a destination location of the route using the crowdsourcing data; anddetermining a route based at least in part on a predicted fuelconsumption of the route using the crowdsourcing data. The determininginformation corresponding to a route of interest to at least one vehiclefurther can include monitoring a distance traveled by the at least onevehicle after reaching a destination, and predicting availability ofparking spaces at the destination based at least in part on the distancetraveled; and monitoring an amount of time traveled by the at least onevehicle after reaching a destination, and predicting availability ofparking spaces at the destination based at least in part on the amountof time traveled. The determining information corresponding to a routeof interest to at least one vehicle further comprises: measuring a timetaken to travel a predefined percent of the route until the at least onevehicle comes to a halt at a predetermined location; and predicting anaverage amount of time used to find parking at the predeterminedlocation using the time taken to travel a predefined percent of theroute. The determining information corresponding to a route of interestto at least one vehicle further comprises at least one of: determiningpopularity of a fueling station along the route; determining type offuel sold at the fueling station along the route; determining popularityof a business along the route; and determining popularity of a rest areaalong the route.

Crowd-Sourced Map Updating and Obstacle Annotating

Next, a system to crowd-source the updates of precision maps with datafrom smart vehicles is detailed. In embodiments, crowd-sourced obstacledata can be used to update a map with precision. The obstacles can berocks, boulders, pot-holes, manhole, utility hole, cable chamber,maintenance hole, inspection chamber, access chamber, sewer hole,confined space or can be water pool or rising tidal waves that affectthe road as detected by a plurality of vehicles. Such crowd-sourcedinformation is updated into the map and annotated by time, weather andperiodicity. The detected obstacle information may include a geographiclocation of the vehicle and a predetermined map of the road. Thecomputer system may determine the geographic location of the obstacleby, for example, using a laser rangefinder or light detection andranging (LIDAR) unit to estimate a distance from the obstacle to the atleast two objects near the vehicle and determining the geographiclocation of the obstacle using triangulation, for example. Suchinformation is updated into the map system and marked as temporal.During use, if recent vehicles take defensive driving around thetemporary obstacle, the map adds the obstacles to the map for the routeguidance module to advise vehicles. If recent vehicles drive the road asthough the obstacle does not exist, the system removes the obstacle fromthe map database, but keeps track of the history in case it is aperiodic obstacle. The obstacle information is also reported togovernment agency for repair/maintenance.

In another embodiment, if vehicles drive through the lane with a smoothline or curve, but abruptly brakes, the system infers that the road hasdefects or potholes, for example, and the bad infrastructure is reportedfor path planning (to add more travel time, or to change the route toavoid the bad road infrastructure if it is long.

The new information is used to update a digital map that lacks thecurrent information or that contains inaccuracies or may be incomplete.The digital map stored in the map database may be updated using theinformation processed by a map matching module, matched segment module,and unmatched segment module. The map matching module, once it hasreceived obstacle location and GPS traces, processes obstacle locationsand GPS traces by matching them to a road defined in the digital map.The map matching module matches the obstacles and the GPS traces withthe most likely road positions corresponding to a viable route throughthe digital map by using the processor to execute a matching algorithm.In one example, the matching algorithm may be a Viterbi matchingalgorithm. Where the GPS traces do match a road defined in the digitalmap, the matched trace to which the GPS traces match and obstacleinformation are sent to the matched segment module for furtherprocessing as will be described below. Where the GPS traces do not matcha road defined in the digital map, the unmatched trace to which the GPStraces are correlated with and the obstacle position information aresent to the unmatched segment module for further processing. The matchedsegment module and unmatched segment module both provide metadata to themap updating module. The metadata may include obstacle metadata roadgeometry refinement metadata, road closure and reopening metadata,missing intersection metadata, missing road data and one-way correctionmetadata. The map updating module updates the digital map in the mapdatabase.

The process to update maps using crowd-sourced data may begin with theunmatched segment module clustering the unmatched GPS traces receivedfrom the map matching module. Many available algorithms may be suitablefor this process, but in one example, an agglomerative clusteringalgorithm that iteratively compares GPS traces with each other andcombines those that fall within a pre-determined tolerance into acluster may be used. One example of such and algorithm uses theHausdorff distance as its distance measure in the clustering algorithm.Once the cluster is selected, the unmatched segment module may produce asingle road geometry for a cluster of unmatched GPS traces using acenterline fitting procedure in which the single road geometry describesa new road segment with the obstacle which is not described in thecurrent map database. In one example, a polygonal principal curvealgorithm or a Trace Clustering Algorithm (TC1) algorithm can be used.The digital map can be modified to include the new road, includingpossibly new intersections in the base map and any associated pointersor indices updated.

Flock Navigation

Next a flock control behavior is detailed. In one embodiment, aplurality of cars follow a leader car, who in turn is following a targetvehicle or a target driving plan. The leader, or the first car in thegroup would automatically or manually take evasive actions to avoid anobstacle, and the information is transmitted via vehicle to vehiclecommunication such as DSRC to following vehicles, and the driving pathof the entire flock is adjusted according to the obstacle. “Flocking” isthe collective motion of a large number of self-propelled entities andis a collective animal behavior exhibited by many living beings such asbirds, fish, bacteria, and insects. It is considered an emergentbehavior arising from simple rules that are followed by individuals anddoes not involve any central coordination. The vehicle communicationswould identify vehicles traveling as a flock, and the vehicles performdistributed flocking operation by communication over the wirelessnetwork. One embodiment of the vehicle flocking process has thefollowing structure:

initialise_vehicle_positions( ) LOOP    place_vehicles( )   move_all_vehicles_to_new_positions( ) END LOOP

Each of the vehicles rules works independently, so, for each vehicle,the process calculates how much it will get moved by each of the threerules, generating three velocity vectors. The three vectors to thevehicle's current velocity to work out its new velocity.

PROCEDURE move_all_vehicles_to_new_positions( )    Vector v1, v2, v3   Vehicle b    FOR EACH VEHICLE b       v1 = rule1(b)       v2 =rule2(b)       v3 = rule3(b)       b.velocity = b.velocity + v1 + v2 +v3       b.position = b.position + b.velocity    END

The Vehicles Rules are discussed next. One embodiment simulates simpleagents (vehicles) that are allowed to move according to a set of basicrules. The result is akin to a flock of birds, a school of fish, or aswarm of insects. In one embodiment, flocking behavior for each vehicleis controlled by three rules:

Separation—avoid crowding neighbors (short range repulsion)

Alignment—steer towards average heading of neighbors

Cohesion—steer towards average position of neighbors (long rangeattraction)

Rule 1: Vehicles try to go towards the center of mass of neighboringvehicles. The ‘center of mass’ is simply the average position of all thevehicles. Assume there are N vehicles, called b1, b2, . . . bN. Also,the position of a vehicle b is denoted b.position. Then the ‘center ofmass’ c of all N vehicles is given by: c=(b1.position+b2.position++ . .. bN.position)/N

However, the ‘center of mass’ is a property of the entire flock ofvehicles; it is not something that would be considered by an individualvehicle. Each vehicle is moved toward its ‘perceived center’, which isthe center of all the other vehicles, not including itself. Thus, forvehicleJ (1<=J<=N), the perceived center pcJ is given by:

pcJ=(b1.position+b2.position+ . . . +bJ−1.position+bJ+1.position+ . . .+bN.position)/(N−1)

Having calculated the perceived center, the system moves the vehicletowards it. To move it 1% of the way towards the center this is given by(pcJ−bJ.position)/100 as:

PROCEDURE rule1(vehicle bJ)    Vector pcJ    FOR EACH VEHICLE b       IFb != bJ  THEN pcJ = pcJ + b.position    pcJ = pcJ / N−1    RETURN (pcJ −bJ.position) / 100

Rule 2: Vehicles try to keep a small distance away from other objects(including other vehicles). The rule ensures vehicles don't collide intoeach other. If each vehicle within a defined small distance (say 100units) of another vehicle, the vehicle is moved away. This is done bysubtracting from a vector c the displacement of each vehicle which isnear by.

   PROCEDURE rule2(vehicle bJ)          Vector c = 0;          FOR EACHVEHICLE b           IF b != bJ THEN             IF |b.position −bJ.position| < 100 THEN c = c − (b.position − bJ.position)         RETURN c

If two vehicles are near each other, they will be slightly steered awayfrom each other, and at the next time step if they are still near eachother they will be pushed further apart. Hence, the resultant repulsiontakes the form of a smooth acceleration. If two vehicles are very closeto each other it's probably because they have been driving very quicklytowards each other, considering that their previous motion has also beenrestrained by this rule. Suddenly jerking them away from each other isnot comfortable for passengers and instead, the processes have them slowdown and accelerate away from each other until they are far enough apartfor our liking.

Rule 3: Vehicles try to match velocity with near vehicles.

This is similar to Rule 1, however instead of averaging the positions ofthe other vehicles we average the velocities. We calculate a ‘perceivedvelocity’, pvJ, then add a small portion (about an eighth) to thevehicle's current velocity.

PROCEDURE rule3(vehicle bJ)    Vector pvJ    FOR EACH VEHICLE b       IFb != bJ THEN          pvJ = pvJ + b.velocity       END IF    END    pvJ= pvJ / N−1    RETURN (pvJ − bJ.velocity) / 8 END PROCEDURE

Additional rules is implemented as a new procedure returning a vector tobe added to a vehicle's velocity.

Action of a crowd or traffic is discussed next. For example, to handlestrong traffic.

PROCEDURE strong_traffic(Vehicle b)    Vector traffic    RETURN trafficEND PROCEDURE

This function returns the same value independent of the vehicle beingexamined; hence the entire flock will have the same push due to thetraffic or crowd.

Limiting the speed of vehicles is discussed next. For a limiting speedvlim:

PROCEDURE limit_velocity(Vehicle b)   Integer vlim   Vector v   IF|b.velocity| > vlim THEN     b.velocity = (b.velocity / |b.velocity|) *vlim   END IF END PROCEDURE

This procedure creates a unit vector by dividing b.velocity by itsmagnitude, then multiplies this unit vector by vlim. The resultingvelocity vector has the same direction as the original velocity but withmagnitude vlim.

The procedure operates directly on b.velocity, rather than returning anoffset vector. It is not used like the other rules; rather, thisprocedure is called after all the other rules have been applied andbefore calculating the new position, ie. within the proceduremove_all_vehiclesto_new_positions:

b.velocity = b.velocity + v1 + v2 + v3 + ... limit_velocity(b)b.position = b.position + b.velocity

Bounding the position is discussed next. In order to keep the flockwithin a certain zone so that they can drive out of them, but thenslowly turn back, avoiding any harsh motions.

PROCEDURE bound_position(Vehicle b)    Integer Xmin, Xmax, Ymin, Ymax,Zmin, Zmax    Vector v    IF b.position.x < Xmin THEN v.x = 10      ELSE IF b.position.x > Xmax THEN v.x = −10    IF b.position.y <Ymin THEN v.y = 10       ELSE IF b.position.y > Ymax THEN v.y = −10   IF b.position.z < Zmin THEN v.z = 10       ELSE IF b.position.z >Zmax THEN v.z = −10    RETURN v

Here of course the value 10 is an arbitrary amount to encourage them todrive in a particular direction.

During the course of flock control, one may want to break up the flockfor various reasons. For example the introduction of a predator maycause the flock to scatter in all directions. The predator can be anobject on an impending collision course with the flock. Scattering theflock can be done. Here the flock can disperse; they are not necessarilymoving away from any particular object, but to break the cohesion (forexample, the flock encounters a dangerously driven vehicle). Thus thesystem negates part of the influence of the vehicles rules.

PROCEDURE move_all_vehicles_to_new_positions( )   FOR EACH VEHICLE b    v1 = m1 * rule1(b)     v2 = m2 * rule2(b)     v3 = m3 * rule3(b)    b.velocity = b.velocity + v1 + v2 + v3 + ...     b.position =b.position + b.velocity

When the risk of collision arises, the process can make m1 negative toscatter the flock. Setting m1 to a positive value again will cause theflock to spontaneously re-form.

Tendency away from a particular place is handled next. If the flock isto continue the flocking behavior but to move away from a particularplace or object (such as a car that appears to collide with the flock),then we need to move each vehicle individually away from that point. Thecalculation required is identical to that of moving towards a particularplace, implemented above as tend_to_place; all that is required is anegative multiplier: v=−m*tend_to_place(b).

The vehicles can be organized into a V formation (sometimes called askein) is the symmetric V-shaped formation for Drag Reduction and FuelSaving where all the cars except the first drive in the upwash from thewingtip vortices of the car ahead. The upwash assists each car insupporting its own weight in flight, in the same way a glider can climbor maintain height indefinitely in rising air.

A flock of automatically driven motor vehicles is detailed next, eachhaving the flock behavior. The motor vehicles of the flock establishes atarget motor vehicle which will be used as a reference for flocking. Theleading motor vehicle of the flock is established as the target motorvehicle by the motor vehicles of the flock. The target motor vehicle maybe established before the motor vehicle start running in flock. Inanother embodiment, the first motor vehicle of the flock detects apreceding motor vehicle with the information from the radar or the CCDcamera on the leading motor vehicle or flock leader, and automaticallyestablishes the detected preceding motor vehicle as a new target motorvehicle. By successively changing new target motor vehicles in thismanner, new motor vehicles may automatically be added to the flock. Evenif a motor vehicle is incapable of communication between motor vehicles,that motor vehicle may be established as a target motor vehicleaccording to an algorithm described later on.

In one embodiment, the leading motor vehicle of the flock establishes ahypothetical target motor vehicle, and transmits items of information ofthe hypothetical target motor vehicle to the other motor vehicles of theflock which follow the flock leader through the inter-vehicularcommunications such as DSRC.

Each vehicle in the flock is responsible for generating a speed planwhich governs the relationship between the position in which the motorvehicle runs and the speed at which the motor vehicle runs. The vehiclesperform determining, based on the speed plan, a planned position to bereached from the present position of the motor vehicle after apredetermined time t, e.g., 1.5 seconds, and a planned speed of themotor vehicle at the planned position in the flock. According to thisfunction, if the speed plan from the present position of the motorvehicle is generated such that the motor vehicle is to maintain thespeed of 80 km/h, i.e., 22.2 m/sec., then the planned position to bereached after the predetermined time t, e.g., 1.5 seconds, is 33.3 mspaced from the present position down the running path B, and theplanned speed at the planned position to be reached is 80 km/h.

The function as the predicted value calculating means serves todetermine a predicted position and a predicted speed to be reached bythe motor vehicle after the predetermined time t. The predicted positionis calculated from the present position, i.e., the traveled distance,the present speed, and the present acceleration of the motor vehiclewhich are given from the communication module 1, and the predicted speedis calculated from the present speed and the present acceleration of themotor vehicle.

The speed/acceleration of the vehicle, based on which the predictedposition and the predicted speed will be determined, is basicallydetermined from the speedometer. The predicted position and thepredicted speed are determined using the speed and the acceleration ofthe motor vehicle and GPS position.

A distance deviation, i.e., a position error, between a planned positionto be reached by the motor vehicle after the predetermined time t basedon the speed plan and the predicted position, described above, to bereached by the motor vehicle, and a speed deviation, i.e., a speederror, between a planned speed to be reached by the motor vehicle afterthe predetermined time t based on the speed plan and the predictedspeed, described above, to be reached by the motor vehicle aredetermined. These deviations are calculated by subtractions. The targetmotor vehicle may be a flock leader. If, however, the target motorvehicle is not a flock leader, then the flock leader calculates aposition, a speed, and an acceleration of the target motor vehicle usingthe laser radar, GPS, or triangulation of RF signals, for example. Basedon the above control algorithm, the engine throttle valve opening, thetransmission, and the brake of each of plural following motor vehiclesare controlled to control the motor vehicles in a flock. The systemdetects the positional data of the preceding motor vehicle throughinter-vehicular communications or the laser radar, and controls thefollowing motor vehicle in the event that the preceding motor vehicledrops out of a normal control range of the vehicle flock control. Evenwhen a motor vehicle drops out of the normal range of the vehicle flockcontrol, the control algorithm controls a following motor vehicle toincrease its inter-vehicular distance up to such a motor vehicle.Therefore, the vehicle platoon control will not be interrupted even whenone or more motor vehicles drops out of the platoon. If it is known thata group of motor vehicles will travel in platoon or motor vehicles arecounted at a tollgate or the like and the incremental count is indicatedto each motor vehicle to let it recognize its position in the platoon,then it is possible to establish the position i for each of the motorvehicles before they travel in platoon. However, in order to handle asituation where another motor vehicle pulls in between motor vehiclesrunning in platoon or another motor vehicle is added to a front or rearend of a platoon of motor vehicles, the process according to the presentinvention makes it possible for each of the motor vehicles running inflock to recognize its position relative to a target motor vehiclethrough inter-vehicular communications. There are two proceduresavailable for each of the motor vehicles running in flock to recognizeits position relative to a target motor vehicle. The first procedure isapplicable to local inter-vehicular communications by which each of themotor vehicles of the flock can communicate with only those motorvehicles which run immediately in front of and behind the motor vehicle.If the flock leader of a flock is selected as a target motor vehicle,then the target motor vehicle transmits its own positional informationi=0 to a next motor vehicle which immediately follows the target motorvehicle. The following motor vehicle adds 1 to i, producing its ownpositional information i=1, recognizes that it is the second motorvehicle from the target motor vehicle, and transmits its own positionalinformation i=1 to a next motor vehicle which immediately follows thesecond motor vehicle. Having received the positional information i=1,the next immediately following motor vehicle adds 1 to i, producing itsown positional information i=2, recognizes that it is the third motorvehicle from the target motor vehicle, and transmits its own positionalinformation i=2 to a next motor vehicle which immediately follows thethird motor vehicle. In this manner, each of the motor vehicles is ableto recognize its position relative to the target motor vehicle with ameans for counting its position and local inter-vehicularcommunications.

If a target motor vehicle is not the flock leader of a flock and thetarget motor vehicle and the flock leader cannot communicate with eachother through inter-vehicular communications, then the flock leader setsits own positional information to i=1, and transmits the own positionalinformation i=1 to a next motor vehicle which immediately follows thetarget motor vehicle.

According to the present invention, as described above, a longitudinalacceleration correcting quantity of each of the motor vehicles of aflock is determined on the basis of predicted deviations of a positionand a speed that are predicted after a predetermined time, from a speedplan, and the speed of the motor vehicle is controlled on the basis ofthe determined longitudinal acceleration correcting quantity. Therefore,the motor vehicles can smoothly be controlled to run in flock along arunning path on a road.

A longitudinal acceleration correcting quantity of a motor vehiclefollowing a target motor vehicle is determined on the basis of aninter-vehicular distance between the following motor vehicle and thetarget motor vehicle and a speed difference there-between after apredetermined time, and the speed of the following motor vehicle iscontrolled on the basis of the determined longitudinal accelerationcorrecting quantity. Consequently, the following motor vehicle canautomatically be driven smoothly along a miming path on a road whilereliably keeping a proper inter-vehicular distance between the followingmotor vehicle and the target motor vehicle.

Since the system arrangements on a flock leader and a following motorvehicle of a flock are identical to each other, the flock leader and thefollowing motor vehicle can automatically be driven in a manner to matchthem using slightly different software or program adaptations madetherefor. Therefore, any one of the motor vehicles of the flock maybecome a flock reader or a following motor vehicle.

Each of following motor vehicles of a flock is not only controlled withrespect to a flock leader, but also always monitors an inter-vehiculardistance between itself and a preceding motor vehicle, so that it canincrease the inter-vehicular distance even when a motor vehicle dropsout of the flock. Therefore, it is not necessary to stop controlling thevehicle flock control when a motor vehicle drops out of the flock. Evenwhen a motor vehicle drops out of a flock, the vehicle flock controlsystem does not stop controlling the other motor vehicles to run inflock, and when the motor vehicle that has dropped out returns to theflock, the vehicle flock control system can continuously control themotor vehicles to run in flock. The vehicle flock control system allowsdifferent types of motor vehicles, such as trucks of different lengths,smaller automobiles, larger automobiles, etc., to be mixed in a flock,and can control those motor vehicles to run in flock. Accordingly, thevehicle flock control system according to the present invention iscapable of stably controlling motor vehicles to run in flock on a roaddesigned for motor vehicles to run automatically, and particularly ofcontrolling the speeds of such motor vehicles smoothly.

Lane Marking Visibility Handling

In some embodiments, a lead vehicle identifies lane information that mayinclude lane markings on the road, and the computer system may use oneor more sensors to sense the lane markings For example, the computersystem may use an image-capture device to capture images of the road andmay detect the lane markings by analyzing the images for predeterminedcolors, shapes, and/or brightness levels that are similar to apredetermined color, shape, and/or brightness of the lane markings. Asanother example, the computer system may project a laser onto the roadand may detect the lane markings by analyzing reflections off the roadfor an intensity that is similar to a predetermined intensity of areflection off the lane markings. The computer system may estimate thelocation of the lane based on the sensed lane markings and control thevehicle to follow the lane. The vehicles behind the lead vehicle canthen simply follow the lead vehicle as part of a flock.

At some point, the lead vehicle may determine that the lane informationhas become unavailable or unreliable. For example, severe fog may bepresent and severely affect the lane markings. In other examples, thevehicle may no longer be able to detect the lane markings on the road,the vehicle may detect contradictory lane markings on the road, thevehicle may no longer be able to determine a geographic location of thevehicle, and/or the vehicle may not be able to access a predeterminedmap of the road. Other examples are possible as well. In response todetermining that the lane information has become unavailable orunreliable, the computer system may use at least one sensor to monitorat least one neighboring vehicle, such as a neighboring vehicle in aneighboring lane or a neighboring vehicle behind the vehicle that ispart of the flock. The computer system may then control the vehicle tomaintain a distance between the vehicle and the at least one neighboringvehicle to be at least a predetermined minimum distance and even if thevehicle is unable to rely on the lane information to estimate a locationof the lane on the road, the vehicle may avoid colliding with the atleast one neighboring vehicle.

In other embodiments, the lane information may include a geographiclocation of the vehicle and a predetermined map of the road. Thecomputer system may determine the geographic location of the vehicle by,for example, querying a location server for the geographic location ofthe vehicle. Alternatively, if the predetermined map indicates ageographic location of at least two objects near the vehicle, thecomputer system may determine the geographic location of the vehicle by,for example, using a laser rangefinder or light detection and ranging(LIDAR) unit to estimate a distance from the vehicle to the at least twoobjects near the vehicle and determining the geographic location of thevehicle using triangulation. Other examples are possible as well. In anycase, the computer system may then locate the geographic location of thevehicle on the predetermined map to determine a location of the lanerelative to the geographic location of the vehicle.

In still other embodiments, the lane information may be derived from aleading vehicle that is in front of the vehicle in the lane andcorrelation with other information such as map data and independent laneanalysis to prevent the blind-following-the blind situation. Thecomputer system may estimate a path of the leading vehicle using, forexample, a laser rangefinder and/or a LIDAR unit. Other examples arepossible as well. Once the computer system has estimated the path of theleading vehicle, the computer system may estimate the location of thelane based on the estimated path. For example, the computer system mayestimate the location of the lane to include the estimated path (e.g.,extend by half of a predetermined lane width on either side of theestimated path). Other examples are possible as well.

In some embodiments, the computer system may maintain a predeterminedthreshold for the lane information, and the computer system maydetermine that the lane information has become unavailable or unreliablewhen the computer system detects that a confidence of the laneinformation (e.g., how confident the computer system is that the laneinformation is reliable) is below the predetermined threshold. In someembodiments, the computer system may additionally maintain apredetermined time period for the lane information, and the computersystem may determine that the lane information has become unavailable orunreliable when the computer system detects that a confidence of thelane information is below the predetermined threshold for at least thepredetermined amount of time.

Upon determining that the lane information has become unavailable orunreliable, the computer system may use at least one sensor to monitorat least one neighboring vehicle. The at least one neighboring vehiclemay include, for example, a neighboring vehicle in a lane adjacent tothe lane in which the vehicle is traveling. As another example, the atleast one neighboring vehicle may include a neighboring vehicle behindthe vehicle in the lane in which the vehicle is traveling. As stillanother example, the at least one neighboring vehicle may include afirst neighboring vehicle and a second neighboring vehicle, each ofwhich may be either in a lane adjacent to the lane in which the vehicleis traveling or behind the vehicle in the lane in which the vehicle istraveling. Other examples are possible as well.

When the lane information has become unavailable or unreliable, thecomputer system may control the vehicle to maintain a distance betweenthe vehicle and the at least one neighboring vehicle to be at least apredetermined distance. The predetermined distance may be, for example,a distance determined to be a safe distance and/or a distanceapproximately equal to the difference between a predetermined lane widthand a width of the vehicle. Other predetermined distances are possibleas well.

In order to maintain the distance between the vehicle and the at leastone neighboring vehicle to be at least the predetermined distance, thecomputer system may continuously or periodically use the at least onesensor on the vehicle to monitor the distance between the vehicle andthe at least one neighboring vehicle. The computer system may monitorthe distance between the vehicle and the at least one neighboringvehicle using, for example, a laser rangefinder and/or LIDAR unit. Ifthe distance between the vehicle and the at least one neighboringvehicle becomes less than the predetermined distance, the computersystem may move the vehicle away from the at least one neighboringvehicle in order to maintain the distance between the vehicle and the atleast one neighboring vehicle to be at least the predetermined distance.

In some embodiments, in addition to maintaining the distance between thevehicle and the at least one neighboring vehicle to be at least thepredetermined distance, the computer system may additionally maintainthe distance between the vehicle and the at least one neighboringvehicle to be within a predetermined range of the predetermineddistance. In these embodiments, if the distance between the vehicle andthe at least one neighboring vehicle becomes too large (e.g., no longerwithin the predetermined range of the predetermined distance), thecomputer system may move the vehicle closer to the at least oneneighboring vehicle. This may, for example, prevent the vehicle fromdrifting so far away from the neighboring vehicle that the vehicledrifts into a lane on the opposite side of the vehicle from theneighboring vehicle.

As noted above, in some embodiments the at least one vehicle may includea first neighboring vehicle and a second neighboring vehicle. In theseembodiments, maintaining the distance between the vehicle and the atleast one neighboring vehicle may involve maximizing both a firstdistance between the vehicle and the first neighboring vehicle and asecond distance between the vehicle and the second neighboring vehicle(e.g., such that the vehicle remains approximately in the middle betweenthe first neighboring vehicle and the second neighboring vehicle). Eachof the first distance and the second distance may be at least thepredetermined distance.

In some embodiments, in addition to maintaining the distance between thevehicle and the at least one neighboring vehicle to be at least thepredetermined distance, the computer system may determine an updatedestimated location of the lane. To this end, the computer system may usethe at least one sensor to monitor at least a first distance to the atleast one neighboring vehicle and a second distance to the at least onevehicle. Based on the first distance and the second distance, thecomputer system may determine a first relative position and a secondrelative position (e.g., relative to the vehicle) of the at least oneneighboring vehicle. Based on the first relative position and the secondrelative position, the computer system may estimate a path for the atleast one neighboring vehicle. The computer system may then use theestimated path to determine an updated estimated location of the lane.For example, in embodiments where the at least one neighboring vehicleis traveling in a lane adjacent to the lane in which the vehicle istraveling, the computer system may determine the estimated location ofthe lane to be substantially parallel to the estimated path (e.g., thelane may be centered on a path that is shifted from the estimated pathby, e.g., a predetermined lane width and may extend by half of thepredetermined lane width on either side of the path). As anotherexample, in embodiments where the at least one neighboring vehicle istraveling behind the vehicle in the lane in which the vehicle istraveling, the computer system may determine the estimated location ofthe lane to be an extrapolation (e.g., with constant curvature) of theestimated path. Other examples are possible as well.

In some embodiments, the computer system may additionally use a speedsensor to monitor a speed of the at least one neighboring vehicle andmay modify a speed of the vehicle to be less than the speed of the atleast one neighboring vehicle. This may allow the vehicle to be passedby the at least one neighboring vehicle. Once the at least oneneighboring vehicle has passed the vehicle, the at least one neighboringvehicle may become a leading vehicle, either in a lane adjacent to thelane in which the vehicle is traveling or a leading vehicle that is infront of the vehicle in the lane in which the vehicle is traveling, andthe computer system may estimate the location of the lane of the roadbased on an estimated path of the leading vehicle, as described above.

In some embodiments, the computer system may begin to monitor the atleast one neighboring vehicle only in response to determining that thelane information has become unavailable or unreliable. In theseembodiments, prior to determining that the lane information has becomeunavailable or unreliable, the computer system may rely solely on thelane information to estimate the location of the lane. In otherembodiments, however, the computer system may also monitor the at leastone neighboring vehicle prior to determining that the lane informationhas become unavailable or unreliable. In these embodiments, the computersystem may additionally use the distance to the at least one neighboringvehicle to estimate the location of the lane in which the vehicle istraveling. For example, if the at least one neighboring vehicle istraveling in a lane adjacent to the lane in which the vehicle istraveling, the computer system may determine that the lane does notextend to the at least one neighboring vehicle. As another example, ifthe at least one neighboring vehicle is traveling behind the vehicle inthe lane in which the vehicle is traveling, the computer system maydetermine that the lane includes the at least one neighboring vehicle.Other examples are possible as well. Alternatively, in theseembodiments, prior to determining that the lane information has becomeunavailable or unreliable, the computer system may simply use thedistance to the at least one neighboring vehicle to avoid collisionswith the at least one neighboring vehicle.

Further, in some embodiments, once the vehicle begins to monitor the atleast one neighboring vehicle, the computer system may stop using thelane information to estimate the location of the lane in which thevehicle is traveling. In these embodiments, the computer system may relysolely on the distance to the at least one neighboring vehicle to avoidcollisions with the at least one neighboring vehicle until the laneinformation becomes available or reliable. For example, the computersystem may periodically attempt to obtain updated lane information. Oncethe computer system determines that the lane information has becomeavailable or reliable, the lane information has become available orreliable, the computer system may once again rely on the updatedestimated location of the lane and less (or not at all) on the distanceto the at least one neighboring vehicle. The computer system maydetermine that the updated lane information is reliable when, forexample, the computer system determines that a confidence of the updatedlane information is greater than a predetermined threshold. Thepredetermined threshold may be the same as or different than thepredetermined threshold.

In an embodiment, a driver monitoring units can be configured to includefor example, but not limited to, accelerometer, cameras, gyroscope,magnetometer, and the like sensors. In an embodiment, the accelerometercan include at least one accelerometer for measuring a lateral(sideways), longitudinal (forward and aft) and vertical acceleration inorder to determine whether the driver is operating the vehicle in anunsafe or aggressive manner. For example, excessive lateral accelerationmay be an indication that the driver is operating the vehicle at anexcessive speed around a turn along a roadway. Furthermore, it ispossible that the driver may be traveling at a speed well within theposted speed limit for that area of roadway. However, excessive lateralacceleration, defined herein as “hard turns,” may be indicative ofaggressive driving behavior by the driver and may contribute toexcessive wear on tires and steering components as well as potentiallycausing the load such as a trailer to shift and potentially overturn.

As such, it can be seen that monitoring such driver behavior byproviding feedback and recommendations to the driver during theoccurrence of aggressive driving behavior such as hard turns can improvesafety and reduce accidents. In addition, providing recommendations forsuch aggressive driver behavior can reduce wear and tear on the vehicleand ultimately reduce fleet maintenance costs as well as reduceinsurance costs and identify at risk drivers and driving behavior tofleet managers.

In one aspect, the driver monitoring system may be in data communicationwith an on board diagnostic (OBD) system of the vehicle such as via aport. In some vehicle models, the driver monitoring system is in datacommunication with a controller area network (CAN) system (bus) to allowacquisition of certain driver and vehicle operating parametersincluding, but not limited to, vehicle speed such as via thespeedometer, engine speed or throttle position such as via thetachometer, mileage such as via the odometer reading, seat belt status,condition of various vehicle systems including anti-lock-braking (ABS),turn signal, headlight, cruise control activation and a multitude ofvarious other diagnostic parameters such as engine temperature, brakewear, and the like. The OBD or CAN allows for acquisition of theabove-mentioned vehicle parameters for processing thereby and/or forsubsequent transmission to the server 4106.

In an embodiment, the driver monitoring system may also include a GPSreceiver (or other similar technology designed to track location)configured to track the location and directional movement of the driverin either real-time or over-time modes. As is well known in the art, GPSsignals may be used to calculate the latitude and longitude of a driveras well as allowing for tracking of driver movement by inferring speedand direction from positional changes. Signals from GPS satellites alsoallow for calculating the elevation and, hence, vertical movement, ofthe driver.

In an embodiment, the driver monitoring unit may further include amobile data terminal (MDT) mounted for observation and manipulation bythe driver, such as near the vehicle dash. The MDT can be configured toinclude an operator interface such as a keypad, keyboard, touch screen,display screen, or any suitable user input device and may furtherinclude audio input capability such as a microphone to allow voicecommunications. The driver monitoring unit receives inputs from a numberof internal and external sources. The OBD/CAN bus, which provides datafrom the vehicle's on-board diagnostic system, including engineperformance data and system status information. A GPS receiver provideslocation information. The CDR, XLM, or accelerometers provideinformation regarding the vehicle's movement and driving conditions. Anynumber of other sensors, such as but not limited to, a seat belt sensor,proximity sensor, driver monitoring sensors, or cellular phone usesensors, also provide inputs to the driver monitoring system.

In an embodiment, the driver monitoring system may have any type of userinterface, such as a screen capable of displaying messages to thevehicle's driver or passengers, and a keyboard, buttons or switches thatallow for user input. The system or the user interface may have one ormore status LEDs or other indicators to provide information regardingthe status of the device's operation, power, communications, GPS lock,and the like. Additionally, the LEDs or other indicators may providefeedback to the driver when a driving violation occurs. Additionally,monitoring system may have a speaker and microphone integral to thedevice. In an embodiment, the monitoring system may be self-powered,such as by a battery, or powered by the vehicle's battery and/or powergenerating circuitry. Access to the vehicle's battery power may be byaccessing the power available on the vehicle's OBD and/or CAN bus. Thedriver monitoring system may be self-orienting, which allows it to bemounted in any position, angle or orientation in the vehicle or on thedashboard. In an embodiment, the driver monitoring system determines adirection of gravity and a direction of driver movement and determinesits orientation within the vehicle using this information. In order toprovide more accurate measurements of driver behavior, the presentinvention filters gravitational effects out of the longitudinal, lateraland vertical acceleration measurements when the vehicle is on an inclineor changes its horizontal surface orientation. Driver behavior can bemonitored using the accelerometer, which preferably will be a tri-axialaccelerometer. Acceleration is measured in at least one of lateral,longitudinal and/or vertical directions over a predetermined timeperiod, which may be a period of seconds or minutes. An accelerationinput signal is generated when a measured acceleration exceeds apredetermined threshold.

It will be understood that the present invention may be used for bothfleets of vehicles and for individual drivers. For example, the drivermonitoring system described herein may be used by insurance providers tomonitor, recommend, provide feedback, and adjust insurance rates basedon the driving distance. A private vehicle owner may also use thepresent invention to monitor trip risk such as the driver behavior anduser of the vehicle. For example, a parent may use the system describedherein to monitor a new driver or a teenage driver behavior.

An embodiment of the invention provides real-time recommendations,training, or other feedback to a driver while operating the vehicle. Therecommendations are based upon observed operation of the vehicle and areintended to change and improve driver behavior by identifying improperor illegal operation of the vehicle. The driver monitoring system mayidentify aggressive driving violations. For example, based upon theinputs from an acceleration or CDR, aggressive driving behavior can bedetected, such as exceeding acceleration thresholds in a lateral,longitudinal, or vertical direction, hard turns, hard acceleration orjackrabbit starts, hard braking, and/or hard vertical movement of thevehicle.

Further, in an embodiment, the sensor and camera described herein can beconfigured to communicate with the vehicle entertainment system.Typically, this functionality includes pre-installed software or auser-downloadable application from a network source (such as Apple'siTunes or Google's Android Market). The system functionality may includemapping functions, directions, landmark location, voice-control, andmany other desirable features. When such mobile computing device isplaced within the vehicle then a convenient vehicle entertainment systemassociated with the vehicle can be provided. In an embodiment, a remoteswitch can be used to initiate the vehicle entertainment softwareapplication by communicating with the cameras/sensors located in thevehicle and/or software residing on the mobile computing device. Remoteswitch described herein can include one of a number of well-known remoteswitches that uses wireless or wired technology to communicate withmobile computing device. For example, remote switch may include forexample, but not limited to, a Bluetooth, RF, infrared, or otherwell-known wireless communication technology, or it may be connected viaone or more wires to mobile computing device. The switch may be locatedon any vehicle interior surface, such as on a steering wheel, visor,dashboard, or any other convenient location.

In an embodiment, the cameras or sensors may be placed at any place inthe vehicle, such as for example at four corners of the frontwindshield, in a way that it can directly capture the behaviorparameters of the driver. For example, based on the driver gestures, thecameras can detect finger position to detect that driver is pointing ata particular object or vehicle and searches the internet for thevehicle. Further, in an embodiment, a flexible display film adhesivelysecured on the front windshield. The display can be used controlled by acomputer to display info in a discrete way that may not take driver'seyes off the road and opposing vehicles. In an embodiment, at 4404, thedriver monitoring unit 4102 can be configured to transmit the behaviorparameters of the driver to the server 4106. In an embodiment, thedriver behavior parameters described herein can include for example, butnot limited to, vehicle speed, vehicle accelerations, driver location,seatbelt use, wireless device use, turn signal use, driver aggression,detection of CO2 vapor, detection of alcohol, driver seating position,time, and the like. In an embodiment, at 4406, the server 4106 can beconfigured to transmit the driver behavior parameters to one or moreinsurance providers. In an embodiment, at 4408, the server 4106 can beconfigured to analyze the driver behavior parameters and adjust theinsurance rates for the driver. For example, if the driver is drivingroughly by drinking alcohol then the insurance rate may get decreased.In an embodiment, at 4410, the server 4106 can be configured to matchthe driver behavior preferences with similar or substantially similarpreferences of other drivers. The server 4104 can be configured togenerate action recommendations best matching the behavior of thedriver. In an embodiment at 4412, the server 4106 can be configured toprovide the generated recommendations to the driver. Based on the driverbehavior parameters the sever 4106 provides feedback and recommendationsto the driver, such as to improve the driving skills. Further, in anembodiment, a flexible display film adhesively secured on the frontwindshield. The display can be used controlled by a computer to displayinfo in a discrete way that may not take driver's eyes off the road andopposing vehicles. In an embodiment, at 4414, the server 4106 can beconfigured to frequently monitor the behavior parameters associated withthe driver. Any changes in the behavior parameters can affect theoverall system performance and the driver experience. The server 4106can be configured to frequently monitor and dynamically update theinsurance rate and action recommendations, which in turn helps thedriver for effectively improving the driving skills.

In an embodiment, the driver behavior parameters can be used to providecustomized recommendations to drivers by comparing the driver behaviorparameters with other drivers who has similar or substantially similarbehavior parameters. Unlike conventional system, the server 106 can beconfigured to adaptively generate action recommendations for the driverbased on the behavior parameters. The server 106 can be configured tomatch the behavior parameters of the drivers to similar behaviorparameters of the one or more drivers, such as to provide personalizedaction recommendations to the driver. In an embodiment, therecommendations can be filtered in advance of display. In an embodiment,filtered recommendations may be derived from the sources such as forexample, but not limited to, those sources that have added the datawithin a specified time, from those sources that share specificsimilarities with the sources, those sources that have been preselectedby the driver as relevant, those sources that are selected as friends orfriends of friends, and the like, those sources that are determined toprovide valuable reviews/ratings or are specifically declared to beexperts within the system or by the driver, or those users that haveentered at least a minimum amount of data into the system.

The system can monitor the driver behavior and uses the behavior data tomatch with the behavior data of other sources and provide actionrecommendations to the driver. For example, if the driver behaviorparameter indicates that the user is driving very fast (such as 70 kmph)in school zone where the speed limits should be more than 30 kmph thenthe system can be configured to execute one or more rules and providesuggestions to the driver to slow down the speed. Rules can be derivedfrom, for example, but not limited to, automatic generation machinelearning, automatic generation using a generic algorithm, automaticgeneration using a neutral network, automatic generation using a ruleinference system, data mining, generation using a preset list ofrecommendations, and/or a driver behavior. In an embodiment, the sever106 can be configured to receive the recommendation rules such asunidirectional rules, bidirectional rules, generalized rules includingmulti-way rules, rules among items, rules among sets, rules amongcollections, rules with weight factors, rules with priorities,un-weighted and un-prioritized rules, and the like.

A method 4700 is used for selectively providing insurance information toa service provider, according to embodiments as disclosed herein. Atstep 4702, the driver behavior is monitored. The behavior data caninclude external parameters and/or internal parameters. In anembodiment, the driver behavior data/parameters described herein caninclude for example, but not limited to, vehicle speed, vehicleaccelerations, driver location, seatbelt use, wireless device use, turnsignal use, driver aggression, detection of ethanol vapor, driverseating position, time, and the like. In an embodiment, the behaviordata can be over a period of hours, days, weeks, and so forth. In anembodiment, the behavior data gathering can be continuous, at predefinedintervals, or at random intervals. In accordance with some aspects, datacan be gathered while a vehicle is in operation and at other times(e.g., at two a.m. to determine where the vehicle is parked overnight).In an embodiment, a change to an insurance premium and/or an insurancecoverage is prepared, at 4704. The change is based on one or more of thedriver behavior data, wherein each item of driver behavior data can havea different weight assigned. For example, data gathered related toweather conditions might be given less weight than data gathered relatedto user distractions (e.g., passengers, use of a mobile device whilevehicle is in operation, and so forth). In another example, excessivespeed might be assigned a higher weight than data related to safetyperformance of the vehicle. As such, data with a higher weight can begiven more consideration than data with a lower weight (e.g., dataassigned a higher weight can have a greater impact on the cost ofinsurance). Thus, if the user is traveling at (or below) the speed limitand speed is assigned a greater weight, then the safe speed will tend todecrease (or remain constant) the cost of insurance.

In an embodiment, the driver is notified of the change, at 4706. Thenotification can be in any perceivable format. In an example, thenotification is provided as a dashboard-mounted display. In anotherexample, presenting the change can include displaying the modified costof the insurance policy in a dashboard-mounted display and/or a heads-updisplay. In an embodiment, a service provider is notified of the change,at 708. At substantially the same time as notifying the service provider(or trusted third party) of the change, parameters taken intoconsideration (and associated weight) can also be provided. In such amanner, the service provider (or third party) can selectively furthermodify the cost of insurance, which can be communicated to the userthough the vehicle display or through other means.

The service provider (or third party) might be provided the changeinformation less often than the insurance cost change information isprovided to the user. For example, the user can be provided theinsurance cost change information dynamically and almost instantaneouslywith detection of one or more parameters that can influence theinsurance cost. However, the insurance provider (or third party) mightonly be notified of the change after a specified interval (or based onother intervals). For example, insurance cost changes might beaccumulated over a period of time (e.g., two weeks) and an average ofthe insurance cost changes might be supplied to insurance provider. Insuch a manner, the user has time to adjust parameters that tend toincrease (or decrease) the cost of insurance, which allows the user tohave more control over the cost of insurance.

In an embodiment, Vertical market specialization for insurance isprovided where markets are defined based on granular aspects of coverageand presented to one or more insurance subsystems to obtain quotes for acoverage premium. Such specialization allows insurance companies tocompete in more specific areas of insurance coverage, which allows formore accurate premium rates focused on the specific areas or one or morerelated scenarios. In addition, the granular aspects of coverage can beprovided to one or more advertising systems in exchange for furtherlowered rates, if desired.

According to an example, an insurance market can be defined based ongranular information received regarding an item, a related person, useof the item, etc. Based on the market, premium quotes can be obtainedfrom one or more insurance subsystems related to one or more insurancebrokers. In addition, rates can be decreased where the granularinformation can be provided to an advertising system, in one example. Inthis regard, targeted advertisements can additionally be presented tosystem related to requesting the insurance coverage. Policies can beautomatically selected based on preferences, manually selected using aninterface, and/or the like.

An exemplary system customizes insurance rates to correspond to behaviordriver, according to embodiments as disclosed herein. In an embodiment,the server 4106 can be configured to maintain a database component 4802including data related to different driver behaviors. Such leveragingfrom data banks enables insurance providers to bid in real time, andhence an owner and/or user of a vehicle can benefit from competitionamong various insurance providers, to obtain optimum rates. The serverincludes a rate adjustment component 4804 that in real time candetermine the various rates from a plurality of insurance providers 4110(1 to N, where N is an integer). In one particular aspect, a retrievalagent (not shown) associated with the rate adjustment component 4804 canpull insurance data from the insurance providers based on the contextualdata supplied thereto. For example, such contextual data can be datarecords related to driver behavior, the vehicle 4102 (such as auto shopservice records, current service status for the car, and the like), datarelated to the individual driver (such as health records, criminalrecords, shopping habits, and the like), data related to the environment(road condition, humidity, temperature, and the like) and data relatedto real time driving (frequency of braking, accelerating, intensity ofsuch actions, and the like).

The retrieval agent (not shown) can pull data from the insuranceproviders 4110 and further publish such data to enable a richinteraction between the users on a display or a within a writtencommunication environment. The retrieval agent can further generate aninstance for a connection with the insurance providers. Accordingly, aconnection instance can be employed by the rate adjustment component4804 to store connection information such as the state of dataconveyance, the data being conveyed, connection ID and the like. Suchinformation can additionally be employed to monitor progress of datatransfer to the written communication environment or display, forexample. Accordingly, drivers/owners of motor vehicles can pull orreceive data from the insurance providers 4110, wherein received datacan be posted (e.g., displayed on a monitor) and the connection instancecan be concurrently updated to reflect any successful and/or failed dataretrievals. Thus, at any given moment the connection instance caninclude the most up-to-date version of data transferred between themotor vehicle and the insurance providers. In an embodiment, a switchingcomponent 4806 can be configured to automatically switch user/driver toan insurance provider/company that bids the best rate. Such switchingcomponent 4806 can employ interrupts both in hardware and/or software toconclude the switching from one insurance provider to another insuranceprovider. For example, the interrupt can convey receipt of a moreoptimal insurance rate or completion of a pull request to the insuranceproviders 4110 or that a configuration has changed. In one particularaspect, once an interrupt occurs, an operating system analyzes the stateof the system and performs an action in accordance with the interrupt,such as a change of insurance provider, for example

Such interrupts can be in form of asynchronous external events to theprocessor that can alter normal program flow. Moreover, the interruptscan usually require immediate attention from a processor(s) associatedwith the system. In one aspect, when an interrupt is detected, thesystem often interrupts all processing to attend to the interrupt,wherein the system can further save state of the processor andinstruction pointers on related stacks.

According to a further aspect, the switching component 4804 can employan interrupt dispatch table in memory, which can be accessed by theprocessor to identify a function that is to be called in response to aparticular interrupt. For example, a function can accept a policy froman insurance provider, cancel an existing policy, and/or clear theinterrupt for a variety of other reasons. The function can executeprocesses such as clearing the state of the interrupt, calling a driverfunction to check the state of an insurance policy and clearing, settinga bit, and the like.

The switching component 806 further includes an analyzer component 4902,which further employs threshold ranges and/or value(s) (e.g., pricingranges for insurance policies, terms of the insurance policy, and thelike) according to a further aspect of the present invention. Theanalyzer component 4902 can be configured to compare a received valuefor insurance coverage to the predetermined thresholds, which can bedesignated by an owner/driver. Accordingly, the analyzer component 902can determine if the received insurance coverage policies are within thedesired range as specified by a user an “accept” or “reject”, and/orfurther create a hierarchy from “low” to “high” based on criteriadesignated by the user (e.g., price of the insurance policy, terms ofthe insurance policy, and the like).

According to a further aspect, the analyzer component 4902 can furtherinteract with a rule engine component 4904. For example, a rule can beapplied to define and/or implement a desired evaluation method for aninsurance policy. It is to be appreciated that the rule-basedimplementation can automatically and/or dynamically define and implementan evaluation scheme of the insurance policies provided. Accordingly,the rule-based implementation can evaluate an insurance policy byemploying a predefined and/or programmed rule(s) based upon any desiredcriteria (e.g., criteria affecting an insurance policy such as durationof the policy, number of drivers covered, type of risks covered, and thelike.).

In a related example, a user can establish a rule that can implement anevaluation based upon a preferred hierarchy (e.g., weight) of criteriathat affects the insurance policy. For example, the rule can beconstructed to evaluate the criteria based upon predeterminedthresholds, wherein if such criteria does not comply with setthresholds, the system can further evaluate another criteria orattribute(s) to validate the status (e.g., “accept” or “reject” theinsurance bid and operate the switching component based thereon). It isto be appreciated that any of the attributes utilized in accordance withthe subject invention can be programmed into a rule-based implementationscheme.

A method for customizing insurance rates of a driver in real time isdiscussed next. The method presents information related to a real-timeinsurance rate, according to embodiments as described herein. In anembodiment, at 5102, Metadata can be collected pertaining to real-timeoperation of a vehicle and at least a portion of the metadata can beevaluated, as shown at 5104. The metadata described herein can includedriver behavior data, contextual information, driver history, andreal-time driving information that relates to operation of a driver andvehicle, and the like. Based upon a result of the evaluation, there canbe calculation a real-time insurance rate, such as shown at 5106. In anembodiment, at 5108, determination can be made on how to present thecalculated rate. For example, the determination can be if the rateshould be shown on a center console or a heads-up display. Adetermination can also be made on how to display data (e.g., if anumerical rate should be disclosed or a color element should be lit).Additionally, a determination can be made on other data to disclose,such as safety, environment impact, cost of operating vehicle, a targetspeed, group rank, and the like. The determined rate and otherdetermined data can be presented through a display, such as shown at5110. Thus, the determined rate is presented upon a display viewable tothe driver of the vehicle. In an embodiment, at 5112, the method 5100includes determining if feedback should be presented to the user. Thefeedback can be supplied in real-time as well as be a collective summarypresented after a driving session is complete. If no feedback should bepresented, then the method 5100 can end at 5114. In one instance, ifthere is a new driver attempting to obtain a full drivers license (e.g.,teenage driver) or newer driver, then the check 5112 can determinefeedback should be automatically provided. In another embodiment, anoperator can be solicited on if feedback should be presented dependingon a response the method 5100 can end or continue.

Operation of the vehicle and driver can be evaluated at 5116, which canoccur though different embodiments. As a user operates a vehicle,metadata can be collected and evaluated in real-time. In an alternativeembodiment, data can be collected, but evaluation does not occur untilthe check 5112 determines feedback should be presented. At 5118, therecan be determining feedback for suggesting future driving actions forthe operator to perform in future driving to lower the insurance rate.The method 5100 can include presenting the feedback (e.g., through thedisplay, through a printout, transferring feedback as part of e-mail ora text message, etc.) at 5120. The feedback can be directly related to adriving session as well as is an aggregate analysis of overall drivingperformance (e.g., over multiple driving sessions).

In an embodiment, at 5202, an on-board monitoring system (such as drivermonitoring unit) 4102 is installed in a vehicle to facilitate thecollection of real-time data from the vehicle and forwarding of thereal-time data to an insurance provider. At 5204, the on-boardmonitoring system can be associated with the on-board data/diagnosticcontrol units and system(s) incorporated into the vehicle. The on-boarddata/diagnostic control units and system(s) can include the vehiclesengine control unit/module (ECU/ECM), transmission control unit (TCU),power train control unit (PCU), on-board diagnostics (OBD), sensors andprocessors associated with the transmission system, and other aspects ofthe vehicle allowing the on-board monitoring system to gather sufficientdata from the vehicle for a determination of how the vehicle is beingdriven to be made. The on-board monitoring system can be communicativelycoupled by hard wiring to the on-board diagnostic system(s) or thesystems can be communicatively associated using wireless technologies.In an embodiment, at 5206, a mobile device (e.g., a cell phone) can beassociated with the onboard monitoring system where the mobile devicecan facilitate communication between the on-board monitoring systemswith a remote insurance provider system. The mobile device providesidentification information to the on-board monitoring system to beprocessed by the on-board monitoring system or forwarded an insuranceprovider system to enable identification of the driver. In anembodiment, at 5208, communications are established between the on-boardmonitoring system and the mobile device with the remote insuranceprovider system. In one embodiment it is envisaged that the on-boardmonitoring system and the insurance provider system are owned andoperated by the same insurance company. However, the system could beless restricted whereby the insurance provider system is accessible by aplurality of insurance companies with the operator of the on-boardmonitoring system, e.g., the driver of the vehicle to which the on-boardmonitoring system is attached, choosing from the plurality of insuranceproviders available for their particular base coverage. In such anembodiment, upon startup of the system the insurance provider system candefault to the insurance company providing the base coverage and theoperator can select from other insurance companies as they require. Overtime, as usage of the on-board monitoring system continues, at 5210,there is a likelihood that various aspects of the system might need tobe updated or replaced, e.g., software update, hardware updates, etc.,where the updates might be required for an individual insurance companysystem or to allow the on-board monitoring system to function with oneor more other insurance company systems. Hardware updates may involvereplacement of a piece of hardware with another, while software updatescan be conducted by connecting the mobile device and/or the on-boardmonitoring system to the internet and downloading the software from acompany website hosted thereon. Alternatively, the software upgrade canbe transmitted to the mobile device or the on-board monitoring system bywireless means. As a further alternative the updates can be conferred tothe mobile device or the on-board monitoring system by means of aplug-in module or the like, which can be left attached to the respectivedevice or the software can be downloaded there from.

Monitoring can employ components of an on-board monitoring system,mobile device components, e.g., cell phone system, or any other systemcomponents associated with monitoring the vehicle as it is being driven.Such components can include a global positioning system (GPS) todetermine the location of the vehicle at any given time, such a GPS canbe located in a cell phone, as part of the on-board monitoring system,or an external system coupled to the monitoring system/cell phone—suchan external system being an OEM or after sales GPS associated with thevehicle to be/being driven. A video data stream can be gathered from avideo camera coupled to the on-board monitoring system recording theroad conditions, etc. throughout the journey. Information can also begathered from monitoring/control system(s) that are integral to thevehicle, e.g., the vehicle's engine control unit/module (ECU/ECM) thatmonitors various sensors located throughout the engine, fuel and exhaustsystems, etc. In an embodiment, at 5304, the dynamically gathered data(or driver behavior data) is transmitted to an insurance evaluationsystem. In an embodiment, at 5306, the gathered data is analyzed. Suchanalysis can involve identifying the route taken by the driver, thespeed driven, time of day the journey was undertaken, weather conditionsduring the journey, other road traffic, did the user use their cellphone during the journey?, and the like. In an embodiment, at 5308, thegathered data is assessed from which an insurance rate(s) can bedetermined. For example, if the driver drove above the speed limit thenan appropriate determination could be to increase the insurance premium.In an embodiment, at 5310, the driver can be informed of the newlydetermined insurance rate. Any suitable device can be employed such asinforming the user by cell phone, a display device associated with theon-board monitoring system, or another device associated with thevehicle. The information can be conveyed in a variety of ways, includinga text message, a verbal message, graphical presentation, change oflight emitting diodes (LED's) on a display unit, a HUD, etc. At 5312,the driver can continue to drive the vehicle whereby the method canreturn to 5302 where the data gathering is commenced once more.Alternatively, in an embodiment, at 5312, the driver may complete theirjourney and data gathering and analysis is completed. In an embodiment,at 5314 the driver can be presented with new insurance rates based uponthe data gathered while they were driving the vehicle. The new insurancerates can be delivered and presented to the driver by any suitablemeans, for example the new insurance rates and any pertinent informationcan be forwarded and presented to the driver via a HUD employed as partof the real time data gathering system. By employing a HUD instantaneousnotifications regarding a change in the driver's insurance policy can bepresented while mitigating driver distractions {e.g., line of sightremains substantially unchanged). Alternatively, the on-board monitoringsystem can be used, or a remote computer/presentation device coupled tothe real time data gathering system where the information is forwardedto the driver via, e.g., email. In another embodiment, the driver canaccess a website, hosted by a respective insurance company, where thedriver can view their respective rates/gathered information/analysissystem, etc. Further, traditional means of communication such as aletter can be used to forward the insurance information to the driver.

Cameras can be used to capture traffic information, according toembodiments as disclosed herein. In an embodiment, at 5402, the method5400 includes mounting cameras on the car to monitor the trafficinformation. For example, the car may include cameras mounted to captureviews in the rearward, downward, and the like directions, on the uppersurface at the leading end of the front portion thereof. The positionfor mounting the cameras is not limited to the left side, right side,upper surface, front side, back side, and the like. For example, if thecar has a left side steering wheel, the camera may be mounted on a rightupper surface at a leading end of the front portion of the car. Thecameras may have an angle of view of about 60, 90, 180, and 360 degree.With the construction, since the camera is mounted for a view in therearward and downward directions on the front portion of the car, it cancapture a wide area of the surface of the road in the vicinity of thedriver's car, and an area in the vicinity of the left front wheel.Furthermore, the camera can also capture a part of the body of the carin the vicinity of the front wheel. Thereby, the relation between thecar and the surface of the road can be recorded. In an example, thecameras can be configured to capture images of the road views includingpotential collision events such as how close car is following car infront, how often brake is used in period of time, hard brakes count moreto reduce driver rating, how frequently does car come close to objectsand obstructions (such as trees, cars on the other direction and cars insame direction) while moving. In an embodiment, at 5404, the method 5400includes receiving the recorded information from the camera and useimage processing techniques to process the information. For example, thesystem uses image processing techniques to determine potential collisionevents such as how close car is following car in front, how often brakeis used in period of time, hard brakes count more to reduce driverrating, how frequently does car come close to objects and obstructions(such as trees, cars on the other direction and cars in same direction)while moving. In an embodiment, at 5502, the method 5500 includesmounting cameras on the car to monitor the driver behavior. The positionfor mounting the cameras is not limited to the left side, right side,upper surface, front side, back side, and the like. The cameras may havean angle of view of about 60, 90, 180, and 360 degree. For example, thecamera can capture driver behavior such as for example, but not limitedto, images of texting and use of phone while driving, speech of drivershouting or cursing at other drivers or other occupants, indications ofintoxication, sleepiness, alcohol level, mood, aggressiveness, and thelike. In an embodiment, at 5504, the method 5500 includes receiving therecorded information from the camera and use image processing techniquesand voice reorganization techniques to process the information. Forexample, the system uses image processing techniques to determine thedriver activity such as whether the driver is using mobile phone whiledriving. In another example, the system uses voice recognitiontechniques to determine the use voice, text, aggressiveness, and thelike. In an embodiment, the item-centric approach determines that manydrivers having similar behavior and the driver who performs activity-Awill also perform activity-B. This has proven to be fairly effective. Onthe other hand, many insurance providers interact with driversonline/offline. Such interaction can produce a stream of contextualinformation that recommendation engines can use. Early systems werebatch oriented and computed recommendations in advance for each driver.Thus, they could not always react to a driver's most recent behavior.Recommendation engines work by trying to establish a statisticalrelationship between drivers and activities associated with therebehavior. The system establishes these relationships via informationabout driver's behavior from vehicle owner, monitoring devices, sensors,and the like. In an embodiment, the recommender systems collect data viaAPIs, insurance application, insurance databases, and the like sources.The insurance sources can be available through social networks, ad hocand marketing networks, and other external sources. For example, datacan be obtained from insurance sites, insurance providers, driverinsurance history, and search engines. All this enables recommendationengines to take a more holistic view of the driver. The recommendationengine can recommend different insurance products that save money forthe driver, or alternatively can even recommend different insurancecompanies to save money. Using greater amounts of data lets the enginesfind connections that might otherwise go unnoticed, which yields bettersuggestions. This also sometimes requires recommendation systems to usecomplex big-data analysis techniques. Online public profiles andpreference listings on social networking sites such as Facebook adduseful data.

Most recommendation engines use complex algorithms to analyze driverbehavior and suggest recommended activities that employ personalizedcollaborative filtering, which use multiple agents or data sources toidentify behavior patterns and draw conclusions. This approach helpsdetermine that numerous drivers who have same or similar type ofbehavior in the past may have to perform one or more similar activitiesin the future. Many systems use expert adaptive approaches. Thesetechniques create new sets of suggestions, analyze their performance,and adjust the recommendation pattern for similar behavior of drivers.This lets systems adapt quickly to new trends and behaviors. Rules-basedsystems enable businesses to establish rules that optimizerecommendation performance.

FIG. 14 is a diagram 5600 illustrates generally, a first vehicle programcommunicating with a second vehicle program through an Inter-Vehiclenetworking, according to embodiments as disclosed herein. In anembodiment, the system develops inter-vehicular networking, computing,transceivers, and sensing technologies in the vehicles. Such vehicleshave embedded computers, GPS receivers, short-range wireless networkinterfaces, and potentially access to in-car sensors and the Internet.Furthermore, they can interact with road-side wireless sensor networksand sensors embedded in other vehicles. These capabilities can beleveraged into distributed computing and sensing applications overvehicular networks for safer driving, dynamic route planning, mobilesensing, or in-vehicle entertainment. The system can includevehicular-specific network protocols, middleware platforms, and securitymechanisms to process the data. As shown in FIG. 14, a first driveroperating a vehicle observes a second driver operating a vehicle withinhis visual range and wants to send a message to the second driver. Thevehicle can include identifying information that is visuallyascertainable such as the model, vehicle color, number of doors, licenseplate number and state. The vehicle may include additional informationthat is only ascertainable from up close or at certain angles, or viacertain technologies, such as a roof top identification number, vehicleidentification number, taxi badge number, Bluetooth, or RFID code, andthe like. In an embodiment, a sender having access to the vehiclemonitoring device and viewing a second vehicle desires to contact thedriver of the second vehicle. In one embodiment, in case of an accidentas detected by an accelerometer or airbag deployment, both vehiclesautomatically exchange insurance information and the drivers simplyconfirm and signs to accept. In another embodiment, in case of ahit-and-run, the vehicle computer would automatically capture insuranceinformation from the other vehicle and store all parameters arising fromthe accident for accident investigator's review. In another embodiment,if one vehicle detects that the other vehicle has a low insurancerating, the vehicle automatically enters a defensive driving mode aroundthat vehicle. As best shown in FIG. 16, the sender initiatescommunication via a telephone or handheld computer or vehicle monitoringdevice and accesses the interface to the inter-vehicle networkingservice and database. The sender can select “send message” from thegraphical or audio menu to send message or directly communicate with thedriver of the second vehicle.

For example, the sender can directly communicate with the driver usingthe inter-vehicle networking or the sender can choose from a table ofmessages that can be sent to the driver using the inter-vehiclenetworking. For example, the message can take the form of voice, audio,video, or other data which can be converted to a digital signal and sentto any communications terminal. The inter-vehicle networking databasereceives the message or encrypted message and reconstructs the message,including the address information. The inter-vehicle networking thenseparates out the address information including such as for example, butnot limited to, license plate number, vehicle identification number, andthe like.

In an embodiment, the message may include a return address for thesender, so that a reply can be returned merely by hitting the “reply to”or “call back” button on the message. One skilled in the art would alsorecognize that the message could be sent anonymously or by anon-returnable address. Alternatively, the message could be a generalbroadcast sent by a police officer or other official sending a warningmessage to speeders or an informational message such as “road closedahead” or other message.

In this case, the transceiver can be a WiMAX system. In anotherembodiment, the transceiver can be a meshed 802 protocol networkconfiguration with a constantly morphing mobile mesh network that helpsdrivers avoid accidents, identify traffic jams miles before theyencounter them, and act as a relay point for Internet access. In oneembodiment, the mesh network can be the ZigBee mesh network. In anotherembodiment, the mesh network can be a modified Wi-Fi protocol called802.11p standard for allowing data exchange between moving vehicles inthe 5.9 GHz band. 802.11p operates in the 5.835-5.925 GHz range, dividedinto 7 channels of 10 MHz each. The standard defines mechanisms thatallow IEEE 802.11™ technology to be used in high speed radioenvironments typical of cars and trucks. In these environments, the802.11p enhancements to the previous standards enable robust andreliable car-to-car and car-to-curb communications by addressingchallenges such as extreme Doppler shifts, rapidly changing multipathconditions, and the need to quickly establish a link and exchange datain very short times (less than 100 ms). Further enhancements are definedto support other higher layer protocols that are designed for thevehicular environment, such as the set of IEEE 1609™ standards forWireless Access in Vehicular Environments (WAVE). 802.11p supportsIntelligent Transportation Systems (ITS) applications such ascooperative safety, traffic and accident control, intersection collisionavoidance, and emergency warning.

One variation of 802.11p is called the Dedicated Short RangeCommunications (DSRC), a U.S. Department of Transportation project aswell as the name of the 5.9 GHz frequency band allocated for the ITScommunications. More information on the 802.11p standard can be obtainedfrom the IEEE. DSRC itself is not a mesh. It's a broadcast, so it onlyreaches vehicles within range. Meshing requires a lot moresophistication. There's a routing aspect to it, relaying messages toother nodes. DSRC is much simpler.

One embodiment uses high-powered, heavily encrypted Wi-Fi thatestablishes point-to-point connections between cars within a half-mileradius. Those connections are used to communicate vital informationbetween vehicles, either triggering alerts to the driver or interpretedby the vehicle's computer. An intelligent car slamming on its brakescould communicate to all of the vehicles behind it that it's coming torapid halt, giving the driver that much more warning that he too needsto hit the brakes.

But because these cars are networked—the car in front of one vehicle isconnected to the car in front it and so forth—in a distributed mesh, anintelligent vehicle can know if cars miles down the road are slamming ontheir brakes, alerting the driver to potential traffic jams. Givenenough vehicles with the technology, individual cars become nodes in aconstantly changing, self-aware network that can not only monitor what'sgoing on in the immediate vicinity, but across a citywide traffic grid.

In one embodiment, the processor receives travel routes and sensor datafrom adjacent vehicles, such information is then used for preparingvehicular brakes for a detected turn or an anticipated turn fromadjacent vehicles. The travel routes can be transmitted over a vehicularWi-Fi system that sends protected information to nearby vehiclesequipped with Wi-Fi or Bluetooth or ZigBee nodes. In one embodiment, amesh-network is formed with Wi-Fi transceivers, wherein each vehicle isgiven a temporary ID in each vehicular block, similar to a cellularblock where vehicles can join or leave the vehicular block. Once thevehicle joins a group, travel routes and sensor data is transferredamong vehicles in a group. Once travel routes are shared, the processorcan determine potential or desired actions from the adjacent vehiclesand adjust appropriately. For example, if the car in front of thevehicle is about to make a turn, the system prepares the brakes andgently tugs the driver's seat belt to give the drive notice that the carin front is about to slow down. In another example, if the processordetects that the driver is about to make a lane change to the left basedon sensor data and acceleration pedal actuation, but if the processordetects that the vehicle behind in the desired lane is also speeding up,the system can warn the driver and disengage the lane change to avoidthe accident. Thus, the processor receives travel routes and sensor datafrom adjacent vehicles and notifying the driver of a detected turn or ananticipated turn from adjacent vehicles. The processor receives travelroutes and sensor data from adjacent vehicles and optimizes groupvehicular speed to improve fuel efficiency. The processor receivestravel routes and sensor data from adjacent vehicles and sequences redlight(s) to optimize fuel efficiency. The processor notifies the driverof driving behaviors from other drivers at a predetermined location. Theprocessor switches turn signals and brakes using a predeterminedprotocol to reduce insurance premium for the driver. The processor warnsthe driver to avoid driving in a predetermined pattern, driving during apredetermined time, driving in a predetermined area, or parking in apredetermined area to reduce insurance premium for the driver. Theprocessor sends driver behavior data to an insurer, including at leastone of: vehicle speed, vehicle accelerations, vehicle location, seatbeltuse, wireless device use, turn signal use, detection of ethanol vapor,driver seating position, and time.

The various systems described above may be used by the computer tooperate the vehicle and maneuver from one location to another. Forexample, a user may enter destination information into the navigationsystem, either manually or audibly. The vehicle may determine itslocation to a few inches based on a combination of the GPS receiverdata, the sensor data, as well as the detailed map information. Inresponse, the navigation system may generate a route between the presentlocation of the vehicle and the destination.

When the driver is ready to relinquish some level of control to theautonomous driving computer, the user may activate the computer. Thecomputer may be activated, for example, by pressing a button or bymanipulating a lever such as gear shifter. Rather than taking controlimmediately, the computer may scan the surroundings and determinewhether there are any obstacles or objects in the immediate vicinitywhich may prohibit or reduce the ability of the vehicle to avoid acollision. In this regard, the computer may require that the drivercontinue controlling the vehicle manually or with some level of control(such as the steering or acceleration) before entering into a fullyautonomous mode.

Once the vehicle is able to maneuver safely without the assistance ofthe driver, the vehicle may become fully autonomous and continue to thedestination. The driver may continue to assist the vehicle bycontrolling, for example, steering or whether the vehicle changes lanes,or the driver may take control of the vehicle immediately in the eventof an emergency.

The vehicle may continuously use the sensor data to identify objects,such as traffic signals, people, other vehicles, and other objects, inorder to maneuver the vehicle to the destination and reduce thelikelihood of a collision. The vehicle may use the map data to determinewhere traffic signals or other objects should appear and take actions,for example, by signaling turns or changing lanes. Once the vehicle hasarrived at the destination, the vehicle may provide audible or visualcues to the driver. For example, by displaying “You have arrived” on oneor more of the electronic displays.

The vehicle may be only partially autonomous. For example, the drivermay select to control one or more of the following: steering,acceleration, braking, and emergency braking.

The vehicle may also have one or more user interfaces that allow thedriver to reflect the driver's driving a style. For example, the vehiclemay include a dial which controls the level of risk or aggressivenesswith which a driver would like the computer to use when controlling thevehicle. For example, a more aggressive driver may want to change lanesmore often to pass cars, drive in the left lane on a highway, maneuverthe vehicle closer to the surrounding vehicles, and drive faster thanless aggressive drivers. A less aggressive driver may prefer for thevehicle to take more conservative actions, such as somewhat at or belowthe speed limit, avoiding congested highways, or avoiding populatedareas in order to increase the level of safety. By manipulating thedial, the thresholds used by the computer to calculate whether to passanother car, drive closer to other vehicles, increase speed and the likemay change. In other words, changing the dial may affect a number ofdifferent settings used by the computer during its decision makingprocesses. A driver may also be permitted, via the user interface, tochange individual settings that relate to the driver's preferences. Inone embodiment, insurance rates for the driver or vehicle may be basedon the style of the driving selected by the driver.

Aggressiveness settings may also be modified to reflect the type ofvehicle and its passengers and cargo. For example, if an autonomoustruck is transporting dangerous cargo (e.g., chemicals or flammableliquids), its aggressiveness settings may be less aggressive than a carcarrying a single driver—even if the aggressive dials of both such atruck and car are set to “high.” Moreover, trucks traveling across longdistances over narrow, unpaved, rugged or icy terrain or vehicles may beplaced in a more conservative mode in order reduce the likelihood of acollision or other incident.

In another example, the vehicle may include sport and non-sport modeswhich the user may select or deselect in order to change theaggressiveness of the ride. By way of example, while in “sport mode”,the vehicle may navigate through turns at the maximum speed that issafe, whereas in “non-sport mode”, the vehicle may navigate throughturns at the maximum speed which results in g-forces that are relativelyimperceptible by the passengers in the car.

The vehicle's characteristics may also be adjusted based on whether thedriver or the computer is in control of the vehicle. For example, when aperson is driving manually the suspension may be made fairly stiff sothat the person may “feel” the road and thus drive more responsively orcomfortably, while, when the computer is driving, the suspension may bemade such softer so as to save energy and make for a more comfortableride for passengers. The adjustment can be done by hand as in thefollowing:

-   -   1. A method to control the car using hand gestures.    -   2. The method of claim 1, further comprising detecting a hand        movement as mouse movement or mouse clicks to control the        vehicle.    -   3. The method of claim 1, further comprising controlling a car        window position using an arm gesture and finger gesture and        exiting by forming a fist.    -   4. The method of claim 1, further comprising controlling a seat        position using an arm gesture and a figure gesture, comprising        moving the seat as specified by an arm angular movement.    -   5. The method of claim 1, further comprising unlocking a hood or        a trunk using an arm gesture and a figure gesture, wherein to        open the trunk a driver makes a right angular movement and an        opposite movement for unlocking the hood.    -   6. The method of claim 1, further comprising controlling        temperature of a driver and front passenger seats using an arm        gesture and a finger gesture, wherein the number of fingers        affect a seat temperature.    -   7. The method of claim 1, further comprising performing a left        arm gesture based navigation control for a car.    -   8. The method of claim 1, further comprising activating a        positioning system and selecting start and destination using        gestures.    -   9. The method of claim 1, further comprising controlling a        mirror position with gesture based control.    -   10. The method of claim 1, further comprising controlling music        using hand gesture.    -   11. The method of claim 1, further comprising controlling        vehicle temperature with a hand gesture.    -   12. The method of claim 1, further comprising controlling sound        volume using a gesture.    -   13. The method of claim 1, further comprising controlling sun        roof opening and closure using a gesture.    -   14. The method of claim 1, further comprising controlling wind        shield wipers with a gesture.    -   15. The method of claim 1, further comprising controlling        operation of a rear view mirror.    -   16. The method of claim 1, further comprising gesturing on a        steering wheel to make calls, receive and respond to texts,        launch apps, get turn-by-turn directions, find the nearest other        local businesses, play music, or search the Internet while        driving.    -   17. The method of claim 1, further comprising controlling an air        conditioning system.    -   18. The method of claim 1, further comprising recognizing a        combination of gestures to create a custom gesture control.    -   19. The method of claim 1, further comprising ultrasonic sensors        to detect gestures.    -   20. The method of claim 1, further comprising rendering a        display or sound or tactile feedback for a detected gesture.

Augmented Reality/Virtual Reality Sports Gaming

FIG. 15 shows an exemplary 360 degree camera on a helmet, for example,for augmenting reality of games. Using augmented reality, various waysmay exist for a user to “participate” in a live event. Generally,augmented reality refers to a presentation of a real world environmentaugmented with computer-generated data (such as sound, video, graphicsor other data). In some embodiments, augmented reality, implemented inconjunction with a live event, may allow a user to control a virtualobject that appears to compete or otherwise interact with theparticipants of the live event. For example, an end user device, such asa mobile phone, tablet computer, laptop computer, or gaming console maybe used to present a live video feed of an event to a user. This livevideo feed may be video of an event that is occurring in real-time,meaning the live event is substantially concurrently with thepresentation to the user (for example, buffering, processing, andtransmission of the video feed may result in a delay anywhere from lessthan a second to several minutes). The presentation of the live eventmay be augmented to contain one or more virtual objects that can be atleast partially controlled by the user. For instance, if the live eventis a stock car race, the user may be able to drive a virtual cardisplayed on the end user device to simulate driving in the live eventamong the actual racers. As such, the user may be able to virtually“compete” against the other drivers in the race. The virtual object, inthis example a car, may be of a similar size and shape to the real carsof the video feed. The user may be able to control the virtual car torace against the real cars present in the video feed. The real carsappearing in the video feed may affect the virtual object. For example,the virtual object may not be allowed to virtually move through a realcar on the augmented display, rather the user may need to drive thevirtual object around the real cars. Besides racing, similar principlesmay be applied to other forms of live events; for example, track andfield events (e.g., discus, running events, the hammer toss, polevaulting), triathlons, motorbike events, monster truck racing, or anyother form of event that a user could virtually participate in againstthe actual participants in the live event. In some embodiments, a usermay be able to virtually replay and participate in past portions of alive event. A user that is observing a live event may desire to attemptto retry an occurrence that happened during the live event. Whileviewing the live event, the user may be presented with or permitted toselect an occurrence that happened in the course of the live event andreplay it such that the user's input affects the outcome of at leastthat portion of the virtualized live event. Using a baseball game as anexample, with runners on first and third, two outs, and the count beingtwo balls and two strikes, the pitcher may throw a splitter,successfully striking out the batter with a pitch in the dirt. Theinning may end and the game may continue. The user may desire to replaythis unsuccessful at-bat with himself controlling the batter during thecommercial break. As such, via an end user device, the user may be ableto indicate the portion of the game he wishes to replay (e.g., the lastat-bat). Game facts from the live event may be used to virtuallyrecreate this at-bat for the user. For instance, the virtual game loadedby the user may use game facts leading up to the at-bat the user hasselected. For instance, the opposing team, the stadium, the score, thetime of day, the batter, the pitcher, and the sequence of pitches thrownby the pitcher may be used to provide the user with a virtual replay ofat least that portion of the baseball game that the user can affect viainput (e.g., swinging and aiming the virtual bat). In replaying theselected portion of the live event, the entire event may be virtualized.As such, referring to the baseball example, the pitcher, stadium, field,fielders, batter, and ball may all be replaced by virtual objects, withone (or more) of the virtual objects, such as the batter, beingcontrolled by the user. As such, this may resemble a video gameinstantiated with data from the live event. In some embodiments, aportion of the live event may involve a playback of a video feed of thelive event with a virtual object that is controlled by the user beingaugmented. Referring again to the example of the baseball game, thepitcher, stadium, fielders, and field may be replayed from the videofeed; the batter and/or ball may be virtualized. As such, the user maycontrol the batter and swing at a virtual ball that has taken the placeof the real ball present in the video feed. Besides baseball, suchreenactment of a portion of a live event may be applied to various formsof sporting events, such as football, soccer, tennis, golf, hockey,basketball, cricket, racing, skiing, gymnastics, and track and fieldevents. Other forms of live events, besides sports, may also bereenacted using such techniques.

FIG. 15A shows a multi-headed camera array 423 that may be at least partof a modular camera system, with each camera forming a module of themodular camera system. The camera array has a flexible structure so thatit is easy to remove a particular camera module from the camera arrayand to add new camera modules to the camera array. The camera modules inthe camera array may be configured in different geometries. For example,the camera array includes multiple camera modules arranged in a line, acylinder, a sphere, or another geometry. Each camera module may beconfigured to point to a different direction so that the camera arraymay capture an object or a scene from multiple directions at the sametime.

The camera system described herein may additionally include a set ofalgorithms for processing the video data captured by the camera array.The set of algorithms are stored on a non-transitory memory forconverting the input across multiple camera modules into a single streamof 3D video (e.g., a single compressed stream of 3D video data). The setof algorithms may be implemented in one or more “modules”. For example,the set of algorithms includes color correction algorithms for smoothingand correcting colors in the video data. In another example, the set ofalgorithms may be implemented in software that stitches the video datafrom multiple cameras into two large-format, panoramic video streams forleft and right eye viewing, and encodes and compresses the video using astandard MPEG format or other suitable encoding/compression format.

The camera array 423 may be constructed using various configurations.For example, the camera modules may be configured in differentgeometries (e.g., a sphere, a line, a cylinder, a cone, a cube, etc.)with the corresponding lenses 113 facing in different directions. Forexample, the camera modules are positioned within the camera array 423in a honeycomb pattern where each of the compartments form an aperturewhere a camera module may be inserted. In another example, the cameraarray 423 includes multiple lenses along a horizontal axis and a smallernumber of lenses on a vertical axis.

In some embodiments, the camera modules in the camera array 423 areoriented around a sphere in different directions with sufficientdiameter and field-of-view to capture enough view disparity to renderstereoscopic images.

The camera array 423 has a flexible structure so that a particularcamera module may be removed from the camera array 423 easily. In someembodiments, the camera modules are rotationally symmetrical such that acamera module may be inserted into the housing, removed, rotated 90degrees, and reinserted into the housing. In this example, the sides ofthe housing may be equidistant, such as a camera module with fourequidistant sides. This allows for a landscape orientation or a portraitorientation of the image frames without changing the base. In someembodiments, the lenses and the camera modules are interchangeable. Newcamera modules may also be added to the camera array 423. In someembodiments, the camera modules in the camera array 423 are positionedto have a sufficient field-of-view overlap so that all objects can beseen by more than one view point. In some embodiments, having the cameraarray 423 configured so that an object may be viewed by more than onecamera may be beneficial for correcting exposure or color deficienciesin the images captured by the camera array 423. Other benefits includedisparity/depth calculations, stereoscopic reconstruction, and thepotential to perform multi-camera high-dynamic range (HDR) imaging usingan alternating mosaic pattern of under- and over-exposure across thecamera array.

In some embodiments, the camera array 423 may also include a microphonearray for capturing sound from all directions. For example, themicrophone array may include a Core Sound Tetramic soundfieldtetrahedral microphone array following the principles of ambisonics,enabling reconstruction of sound from any arbitrary direction. Inanother example, the microphone array includes the Eigenmike, whichadvantageously includes a greater number of microphones and, as aresult, can perform higher-order (i.e. more spatially accurate)ambisonics. The microphone may be mounted to the top of the camera array423, be positioned between camera modules, or be positioned within thebody of the camera array 423. The result can then be rendered as animmersive video and a user can view the video with computer annotationsthereon for augmented reality purposes. In one implementation, the eventmay be a live event, for example, but is not limited to, a footballmatch, a cricket match, a basketball match, a theatre, a concert, andthe like. In one embodiment, the augmented reality content may include,but is not restricted to, live content associated with an event,recorded content associated with an event, a curated content, anadvertising content, or a combination thereof. In another embodiment,the augmented reality content may include, but is not restricted to,information related to a service available at an event, a venue of anevent, a status of a service, or a combination thereof. The system 100may also provide the augmented reality content associated with, but isnot restricted to, a venue of an event, duration of an event, a locationof an event, or a combination thereof, in another implementation.

One embodiment allows combined augmented reality and virtual reality onthe display. The method may include selectively allowing a transmissionof light from a local environment of the user based on a visualizationmode of the display object. The visualization mode may be one of anaugmented reality mode, a virtual reality mode, and a combination ofaugmented and virtual reality modes.

In another embodiment, sensors may be placed to track eye movement aswell as hand gestures and verbal commands. The method may furthercomprise capturing a field-of-view image of each of the user's eyes. Thecaptured field of view image may be used to estimate a head pose of theuser. The captured field-of-view image may be used to convert at leastone physical object to a physically rendered virtual object, and todisplay the physically rendered virtual object to the user. In anotherembodiment, sensors may be placed to track eye movement as well as handgestures and verbal commands. Then, a method comprises tracking amovement of a user's eyes, estimating a depth of focus of the user'seyes based on the tracked eye movement, modifying a light beamassociated with a display object based on the estimated depth of focussuch that the display object appears in focus, and projecting themodified light beam into the user's eyes. The diameter of the projectedlight beam projected to the user's eyes may be less than 0.7 mm.

For the athlete/participant who wish to enhance their gaming viaaugmented or virtual reality, features may include the following:

1. A method for using augmented reality, the method comprising:receiving, by a computerized device, a data stream with a 360 degreeview of a live event on each participant, wherein the data streamcomprises live video augmented with positions of team mates and opposingplayers and recommends a play routine based on live field condition andpositions of other players, wherein the user can select a point of viewfrom a selected participant.

2. The method for using augmented reality of claim 1, wherein the userplays in a virtual reality version of the live event.

3. The method for using augmented reality of claim 1, wherein the liveevent is a sporting event.

4. The method of claim 7, wherein the live event comprises: soccer,football, basketball, tennis, boxing, car racing, golf, ice hockey,badminton, volleyball, cycling, swimming, snooker, martial arts, rugby,motorbike, hockey, table tennis, horse racing, gymnastics, handball,figure skating, wrestling, skiing, diving, skating, archery, sailing,wrestling, fencing, equestrian, rowing, surfing, Beach Volleyball,Pool/Billiards, Lacrosse, Windsurfing, Polo, Tenpin Bowling,Racquetball, Competitive Climbing, Mountain Biking.

FIG. 15 shows an exemplary recommender to aid an athlete in improvingthe game. For example, the process can recommend a strategy in light ofthe opponent's historical performance. In tennis, a player's historicalweakness can be ascertained and a recommendation can be made to optimizesuccess. In a football example, a fourth down module 400 may include aFootball recommender, a Field Goal algorithm, and a Punt algorithm. TheFootball recommender determines the probability of each potential playoutcome associated with the Go For It coaching decision. The Field Goalalgorithm determines the probability of each potential play outcomeassociated with the Field Goal coaching decision. The Punt algorithm1102 determines the probability of each potential play outcomeassociated with the Punt coaching decision. As shown in FIG. 15B, theFootball recommender 402 determines the probability of each potentialplay outcome associated with the Go For It coaching decision on a fourthdown play. The Football recommender 402 receives an expected points (EP)input from the expected points module 300 at block 404, a yards to gain(YTG) for first down input at block 406, and a first down marker (FDM)yard line input at block 408. Preliminary Conversion Rate: At block 410,the Football recommender 402 uses the team's EP value from block 404 andthe YTG distance from block 406 to determine a preliminary first downconversion rate based on historical conversion data. Historical firstdown conversion data is shown in the form of a chart in FIG. 5, whereYTG distances are presented on the x-axis and average first downconversion rates are presented on the y-axis. This historical data showsthat the likelihood of a first down conversion decreases as the YTGdistance increases. Individual lines or equations may be presented toaccount for various EP values. For simplicity, FIG. 5 shows three linesto account for scenarios in which the offense and defense are an equalmatch with the same EP values (NEU), the offense has the advantage (OFFAD), and the defense has the advantage (DEF AD). The historical datapresented in FIG. 5 shows that stronger offenses will convert firstdowns versus weaker defenses (OFF AD) more often than weaker offenseswill convert first downs versus stronger defenses (DEF AD). Similarlines may be provided for specific EP values (e.g., 7-66 points). Bydetermining the first down conversion rate at each YTG distance for eachoffensive match-up, the Football recommender 402 is able to predict thelikelihood of a first down conversion with great precision.

Inside an opponent's 20-yard line (i.e., in the Red Zone), it becomesmore difficult to convert for a first down as the space on the fieldfrom which to work becomes more limited. As the FDM gets closer to theend zone and the YTG distance increases, the challenge of converting afirst down gets progressively more difficult versus similar scenariosoutside of the Red Zone. To account for the challenge of converting afirst down in the Red Zone, the Football recommender 402 may multiplythe preliminary conversion rate by a field position multiplier at block412 based on the YTG distance from block 406 and the FDM yard line fromblock 408 (where 100 represents the opponent's goal line. As an example,take a team that normally has a 50% fourth down conversion rate with 2YTG. If the team faces a fourth down play with 2 YTG outside of the RedZone, the conversion rate may remain at 50%. However, if the team facesa fourth down play with 2 YTG in the Red Zone, such as from theopponent's 2-yard line when the FDM is on the opponent's goal line(FDM=100), the normal 50% conversion rate may be multiplied by thecorresponding field position multiplier of 85.5% to arrive at a loweradjusted conversion rate of 42.7%. The process may adjust team's firstdown conversion rate at block 412 based on particular strengths of histeam. In one embodiment, the Football recommender 402 multiplies theconversion rate by one or more additional multipliers, such as a YTGmultiplier, which may be specified by the coach. As an example, a teamthat thrives on running the football might find that it convertsshort-yardage situations particularly well, because its offense isdesigned to consistently grind out short gains. However, the same teammay have particular difficulty in converting longer-yardage situationsbecause the offense isn't conducive to big plays. In this example, theYTG multiplier may be greater than 100% below 5 YTG to increase theconversion rate in short-yardage situations and less than 100% above 5YTG to decrease the conversion rate in long-yardage situations.Conversely, a team with an explosive offense may be particularlyeffective in converting long yardages but may not have the personnel toget short yardage. In this example, the YTG multiplier may be less than100% below 5 YTG to decrease the conversion rate in short-yardagesituations and greater than 100% above 5 YTG to increase the conversionrate in long-yardage situations. The Indianapolis Colts were a greatexample of this during much of the Peyton Manning era. They were verydangerous in long-yardage situations due to the quality of their passinggame, but due to a poor running game, they often failed to convert inshort-yardage scenarios. The Football recommender 402 may calculate theprobability of a turnover and defensive touchdown as a function of theEP value from block 404 and the FDM yard line from block 408. Thisprobability may be as low as about 0.1% and as high as about 0.5%. Atblock 414, the Football recommender 402 assigns probabilities to eachpotential conversion outcome. The Football recommender 402 may determinenot only the likelihood of a first down conversion at block 412, butalso how likely the team is to score points if the conversion issuccessful at block 416. After a successful conversion, the team couldget just enough yards to get the first down and still not score anypoints on the drive, or it could score a touchdown on the very same playor a subsequent play of the same drive. Therefore, the Footballrecommender 402 may take into account the potential upside of the driveshould the fourth down play be successful at any field position. Atblock 416, the Football recommender 402 uses the team's EP value fromblock 404 and the FDM yard line from block 408 to determine the pointsscored given conversion based on historical scoring data. Historicalscoring data is shown in the form of a chart in FIG. 6, where FDM yardlines are presented on the x-axis (with 0 representing the team's owngoal line and 100 representing the opponent's goal line) and averagepoints scored given conversion are presented on the y-axis. Thishistorical data shows that the likelihood of scoring points increases asthe FDM approaches the opponent's goal line. Individual lines orequations may be presented to account for various EP values. Forsimplicity, FIG. 6 shows three lines to account for scenarios in whichthe offense and defense are an equal match with the same EP values(NEU), the offense has the advantage (OFF AD), and the defense has theadvantage (DEF AD). The historical data presented in FIG. 6 shows thatstronger offenses will score more points versus weaker defenses (OFF AD)than weaker offenses will score versus stronger defenses (DEF AD).Similar lines may be provided for specific EP values (e.g., 7-66points). In this manner, the augmented reality system can enhance thegame.

For viewers who wish to participate via augmented or virtual reality,features may include the following:

1. A method for using augmented reality, the method comprising:receiving, by a computerized device, a data stream with a 360 degreeview of a live event on each participant, wherein the data streamcomprises live video, wherein: the live video comprises a live object;receiving, by the computerized device, input from a user, wherein theinput from the user affects behavior of a virtual object; andpresenting, by the computerized device, the live event augmented by thevirtual object, wherein a behavior of the live object of the live eventaffects the behavior of the virtual object and each participant, whereinthe user can select a point of view from a selected participant.

2. The method for using augmented reality of claim 1, wherein: thevirtual object is presented such that the virtual object appears tocompete with the live object.

3. The method for using augmented reality of claim 1, wherein the liveevent is a sporting event.

4. The method for using augmented reality of claim 1, furthercomprising: receiving, by the computerized device, data corresponding toa second virtual object from a remote computerized device; anddisplaying, by the computerized device, the live event augmented by thevirtual object further augmented with the second virtual object.

5. The method for using augmented reality of claim 4, wherein thebehavior of the second virtual object is affected by a second user.

6. The method for using augmented reality of claim 4, furthercomprising: modifying, by the computerized device, behavior of thevirtual object in response to the second virtual object.

7. A method for using augmented reality, the method comprising:receiving, by a computerized device, data corresponding to a live event;presenting, by the computerized device, the live event up to a point intime; presenting, by the computerized device, a virtual event at leastpartially based on an event that occurred during the live event earlierthan the point in time; receiving, by the computerized device, inputlinked with the virtual event, wherein the input is received from auser; and presenting, by the computerized device, an outcome of thevirtual event, wherein the outcome is at least partially based on theinput received from the user.

8. The method for using augmented reality of claim 7, wherein: thevirtual event is presented at least starting when the live event isstopped.

9. The method of claim 7, wherein the live event is a sporting event.

10. The method of claim 7, wherein the live event comprises: soccer,football, basketball, tennis, boxing, car racing, golf, ice hockey,badminton, volleyball, cycling, swimming, snooker, martial arts, rugby,motorbike, hockey, table tennis, horse racing, gymnastics, handball,figure skating, wrestling, skiing, diving, skating, archery, sailing,wrestling, fencing, equestrian, rowing, surfing, Beach Volleyball,Pool/Billiards, Lacrosse, Windsurfing, Polo, Tenpin Bowling,Racquetball, Competitive Climbing, Mountain Biking.

11. A method for using virtual reality, the method comprising:receiving, by a computerized device, a data stream with a 360 degreeview of a computer generated event on each participant, wherein the datastream comprises live video, wherein: the live video comprises a liveobject; receiving, by the computerized device, input from a user,wherein the input from the user affects behavior of a virtual object;and presenting, by the computerized device, the live event augmented bythe virtual object, wherein a behavior of the live object of the liveevent affects the behavior of the virtual object and each participant.

12. A method for using augmented reality and virtual reality, the methodcomprising: receiving, by a computerized device, a data stream with a360 degree view of a live event on each participant, wherein the datastream comprises live video, wherein: the live video comprises a liveobject; receiving, by the computerized device, input from a user,wherein the input from the user affects behavior of a virtual object;and presenting, by the computerized device, the live event augmented bythe virtual object, wherein a behavior of the live object of the liveevent affects the behavior of the virtual object and each participant,and wherein the virtual reality is rendered by switching the displayfrom an augmented view to a virtual reality view by fading out theaugmented view on the display to show only the virtual reality view andswitching back when augmented reality view is desired.

Moreover, the viewers can collaboratively read the situation andrecommend a strategy in real-time to improve viewer participation. Inthis manner,

1. A method for participating in a game, the method comprising:collecting from viewers of a game one or more state change events duringa game; determining whether a series of the collected state changeevents are a known pattern; requesting, when the series of the collectedstate change events is an unknown pattern, viewers of the game toidentify what caused the collected state change events; and judging, bythe viewers, a best reason among the identified causes of the collectedstate change events.

2. The method of claim 1, comprising running a lottery to decide whichrecommendation is used for the next play in the game.

3. The method of claim 1, further comprising: compensating at least oneviewer who is judged to have the best reason among the identified causesof the collected state change events.

4. The method of claim 1, further comprising: storing as the knownpattern, the best reason among the identified causes of the collectedstate change events when one of the pattern is repeated greater than athreshold number of repeats, and the number of the viewers who agreewith the corresponding best reason is greater than a threshold number ofusers.

5. The method of claim 4, further comprising: associating with thestored best reason a corrective action to be taken in response to afuture corresponding the collected state change events.

6. The method of claim 4, further comprising: displaying to the otherviewers and, players, when the stored best reason is known, theoccurrence of the stored best reason.

7. The method of claim 5, further comprising: transmitting the storedbest reason to other viewers.

8. The method of claim 1, wherein the series of the collected statechange events are at least two state change events that occur within athreshold period of time from each other.

Recognition of Exercise Pattern and Tracking of Calorie Consumption

FIG. 16A illustrates the positions of a ski 126′ and skier 128′ during alofting maneuver on the slope 132′. The ski 126′ and skier 128′ speeddown the slope 132′ and launch into the air 136 at position “a,” andlater land at position “b” in accord with the well-known Newtonian lawsof physics. With an airtime sensor, described above, the unit 10calculates and stores the total airtime that the ski 126′ (and hence theskier 128′) experiences between the positions “a” and “b” so that theskier 128′ can access and assess the “air” time information. Airtimesensors such as the sensor 14 may be constructed with known components.Preferably, the sensor 14 incorporates either an accelerometer or amicrophone. Alternatively, the sensor 14 may be constructed as amechanical switch that detects the presence and absence of weight ontothe switch. Other airtime sensors 14 will become apparent in thedescription which follows. The accelerometer sensesvibration—particularly the vibration of a vehicle such as a ski ormountain bike—moving along a surface, e.g., a ski slope or mountain biketrail. This voltage output provides an acceleration spectrum over time;and information about airtime can be ascertained by performingcalculations on that spectrum. Based on the information, the system canreconstruct the movement path, the height, the speed, among others andsuch movement data is used to identify the exercise pattern. Forexample, the skier may be interested in practicing mogul runs, and thesystem can identify foot movement and speed and height information andpresent the information post exercises as feedback. Alternatively, thesystem can make live recommendations to improve performance to theathlete.

FIG. 16B illustrates a sensing unit 10″ mounted onto a mountain bike138. FIG. 16B also shows the mountain bike 138 in various positionsduring movement along a mountain bike race course 140 (for illustrativepurposes, the bike 138 is shown without a rider). At one location “c” onthe race course 140, the bike 138 hits a dirt mound 142 and catapultsinto the air 144. The bike 138 thereafter lands at location “d”. Asabove, with speed and airtime sensors, the unit 10 provides informationto a rider of the bike 138 about the speed attained during the ridearound the race course 140; as well as information about the airtimebetween location “c” and “d”. In this case, the system can recommend acadence to be reached by the rider, strengthen of abdominals, back andarms, for example.

For golf exercise, It is beneficial to require the golfer to swing thegolf club a plurality of times at each swing position to account forvariations in each swing.. The swing position at which the golf club isswung can be determined by analysis of the measured accelerationprovided by the accelerometer, e.g., the time at which the accelerationchanges. Data obtained during the training stage may be entered into avirtual table of swing positions and estimated carrying distances for aplurality of different swing positions and a plurality of differentswings. A sample format for such a table is as follows, and includes theaveraged carrying distance for each of four different swing positions.The swing analyzer provides a golfer with an excellent estimation of thecarrying distance of a golf ball for a golf club swing at a specificswing position because it has been trained on actual swings by thegolfer of the same club and conversion of information about these swingsinto estimated carrying distances. The golfer can improve their golfgame since they can better select a club to use to hit a golf club fordifferent situations during a round of golf. Also, the swing pattern isused to identify each club path responsible for the curve of any shotand this information is used to improve the golfer. The direction of theclub path relative to the target, out-to-in (fade pattern) or in-to-out(draw pattern), is what I refer to as a players swing pattern. Playersthat swing from in-to-out will tend to hit draws and players that swingfrom out-to-in will tend to hit fades. Where the ball is struck on theface of the driver (strike point) can drastically alter the effect of aplayers swing pattern on ball flight. Thus, the camera detects where theball is struck, and a computer physics model of ball behavior ispresented to the golfer to improve the score. Shots struck off the heelwill tend to fade more or draw less and shots struck off the toe willtend to draw more or fade less. Thus, camera images of the shots struckof heel or toe can also be used to provide patternrecognition/prediction and for training purposes.

For tennis, examples of motions determined for improvement are detailednext. The system can detect if the continental grip is achieved.Throwing Action pattern is also detected, as the tennis serve is anupwards throwing action that would deliver the ball into the air if itwere a baseball pitch. Ball Toss improvements can be determined when theplayer lines the straight arm up with the net post and release the ballwhen your hand reaches eye level. The system checks the forwarddirection so the player can drive weight (and built up momentum) forwardinto the ball and into the direction of the serve.

The sensors can work with a soccer training module with kinematics ofball control, dribbling, passing, crossing, shooting, heading,volleying, taking throw-ins, penalties, corner kicks and free kicks,tackling, marking, juggling, receiving, shielding, clearing, andgoalkeeping. The sensors can work with a basketball training module withkinematics of crossover dribble, behind back, pull back dribble, lowdribble, basic dribble, between legs dribble, Overhead Pass, Chest Pass,Push Pass, Baseball Pass, Off-the-Dribble Pass, Bounce Pass, Jump Shot,Dunk, Free throw, Layup, Three-Point Shot, Hook Shot. The sensors canwork with a baseball training module with kinematics of Hitting,Bunting, Base Running and Stealing, Sliding, Throwing, Fielding GroundBalls, Fielding Fly Balls, Double Plays and Relays, Pitching andCatching, Changing Speeds, Holding Runners, Pitching and PitcherFielding Plays, Catching and Catcher Fielding Plays.

For weight training, the sensor can be in gloves as detailed above, orcan be embedded inside the weight itself, or can be in a smart watch,for example. The user would enter an app indicating that the user isdoing weight exercises and the weight is identified as a dumbbell, acurl bar, and a bar bell. Based on the arm or leg motion, the systemautomatically detects the type of weight exercise being done. In oneembodiment shown in FIG. 15C, with motion patterns captured by glove andsock sensors, the system can automatically detect the followingexemplary exercise:

Upper Body:

-   -   Chest: Barbell Bench Presses, Barbell Incline Presses, Dumbbell        Bench Presses, Dumbbell Incline Presses, Dumbbell Flyes, Cable        Crossovers    -   Back: Pull-Ups, Wide-Grip Lat Pulldowns, One-Arm Dumbbell Rows,        Seated Cable Rows, Back Extensions, Straight Arm Pulldowns    -   Shoulders: Seated Dumbbell Presses, Front Raises, Lateral        Raises, Reverse Flyes, Upright Cable Rows, Upright Barbell Rows

Biceps: Alternate Dumbbell Curls, Barbell Curls, Preacher Curls,Concentration Curls, Cable Curls, Hammer Curls

Triceps: Seated Triceps Presses, Lying Triceps Presses, TricepsKickbacks, Triceps Pushdowns, Cable Extensions, Bench Dips

Lower Body

Quadriceps: Barbell Squats, Leg Presses, Leg Extensions

Hamstrings: Dumbbell Lunges, Straight-Leg Deadlifts, Lying Leg Curls

Calves: Seated Calf Raises, Standing Heel Raises

Abs: Floor Crunches, Oblique Floor Crunches, Decline Crunches, DeclineOblique, Hanging Knee Raises, Reverse Crunches, Cable Crunches, CableOblique Crunches

In one implementation in FIG. 16D, an HMM is used to track weightliftingmotor skills or enthusiast movement patterns. Human movement involves aperiodic motion of the legs. Regular walking involves the coordinationof motion at the hip, knee and ankle, which consist of complex joints.The muscular groups attached at various locations along the skeletalstructure often have multiple functions. The majority of energy expendedduring walking is for vertical motion of the body. When a body is incontact with the ground, the downward force due to gravity is reflectedback to the body as a reaction to the force. When a person stands still,this ground reaction force is equal to the person's weight multiplied bygravitational acceleration. Forces can act in other directions. Forexample, when we walk, we also produce friction forces on the ground.When the foot hits the ground at a heel strike, the friction between theheel and the ground causes a friction force in the horizontal plane toact backwards against the foot. This force therefore causes a breakingaction on the body and slows it down. Not only do people accelerate andbrake while walking, they also climb and dive. Since reaction force ismass times acceleration, any such acceleration of the body will bereflected in a reaction when at least one foot is on the ground. Anupwards acceleration will be reflected in an increase in the verticalload recorded, while a downwards acceleration will be reduce theeffective body weight. Zigbee wireless sensors with tri-axialaccelerometers are mounted to the enthusiast on different body locationsfor recording, for example the tree structure as shown in FIG. 16D. Asshown therein, sensors can be placed on the four branches of the linksconnect to the root node (torso) with the connected joint, left shoulder(LS), right shoulder (RS), left hip (LH), and right hip (RH).Furthermore, the left elbow (LE), right elbow (RE), left knee (LK), andright knee (RK) connect the upper and the lower extremities. Thewireless monitoring devices can also be placed on upper back body nearthe neck, mid back near the waist, and at the front of the right legnear the ankle, among others.

The sequence of human motions can be classified into several groups ofsimilar postures and represented by mathematical models calledmodel-states. A model-state contains the extracted features of bodysignatures and other associated characteristics of body signatures.Moreover, a posture graph is used to depict the inter-relationshipsamong all the model-states, defined as PG(ND,LK), where ND is a finiteset of nodes and LK is a set of directional connections between everytwo nodes. The directional connection links are called posture links.Each node represents one model-state, and each link indicates atransition between two model-states. In the posture graph, each node mayhave posture links pointing to itself or the other nodes.

In the pre-processing phase, the system obtains the human body profileand the body signatures to produce feature vectors. In the modelconstruction phase, the system generate a posture graph, examinefeatures from body signatures to construct the model parameters of HMM,and analyze human body contours to generate the model parameters ofASMs. In the motion analysis phase, the system uses features extractedfrom the body signature sequence and then applies the pre-trained HMM tofind the posture transition path, which can be used to recognize themotion type. Then, a motion characteristic curve generation procedurecomputes the motion parameters and produces the motion characteristiccurves. These motion parameters and curves are stored over time, and ifdifferences for the motion parameters and curves over time is detected,the system then runs the enthusiast through additional tests to confirmthe detected motion.

In one exemplary process for determining exercise in the left or righthalf of the body, the process compares historical left shoulder (LS)strength against current LS strength (3200). The process also compareshistorical right shoulder (RS) strength against current RS strength(3202). The process can compare historical left hip (LH) strengthagainst current LH strength (3204). The process can also comparehistorical right hip (RH) strength against current RH strength (3206).If the variance between historical and current strength exceedsthreshold, the process generates warnings (3208). Furthermore, similarcomparisons can be made for sensors attached to the left elbow (LE),right elbow (RE), left knee (LK), and right knee (RK) connect the upperand the lower extremities, among others.

The system can ask the enthusiast to squeeze a strength gauge,piezoelectric sensor, or force sensor to determine force applied duringsqueeze. The user holds the sensor or otherwise engages the sensor. Theuser then applies and holds a force (e.g., compression, torque, etc.) tothe sensor, which starts a timer clock and triggers a sampling startindicator to notify the user to continue to apply (maximum) force to thesensor. Strength measurements are then sampled periodically during thesampling period until the expiration of time. From the sampled strengthdata, certain strength measurement values are selected, such as themaximum value, average value(s), or values obtained during the samplingperiod. The user can test both hands at the same time, or alternativelyhe may test one hand at a time. A similar approach is used to sense legstrength, except that the user is asked to pushed down on a scale todetermine the foot force generated by the user.

In one embodiment, exercise motion data acquired by the accelerometer ormulti-axis force sensor is analyzed, as will be discussed below, inorder to determine the motion of each exercise stroke during theexercise session (i.e., horizontal vertical or circular). In anotherembodiment for detecting exercise motion using accelerometer, the firstminimum discovered during the scanning is noted as the first xmin andconsidered to be the start of the first brushstroke. The first maximum xvalue following the first minimum x value is located and construed to bethe middle of the first exercise stroke (where exercise motion changesfrom one direction to the other). The next xmin value indicates the endof the first brushstroke and the beginning of the next brushstroke. Thecomputer records the data for each brushstroke and continues on throughthe data to find the next brushstroke, recording each successive motionin memory. For the first brushstroke, the maximum and minimum values ofthe x coordinate (xmax and xmin) are determined. The Y-directionlengths, Ly1 and Ly2, between the data points just before and just aftereach of xmax and xmin (xmax+1, xmax-1, and Xmin+1, xmin-1) are thendetermined. The length Lx along the x axis, between xmax and xmin, isalso determined. Next, if Lx is less than 2 and either Ly1 or Ly2 isgreater than one, then the motion is construed to be vertical. If Ly1and Ly2 are both less than one, then the motion is construed to behorizontal. Otherwise, the motion is construed to be circular.

Data obtained from the gyroscope, if one is used, typically does notrequire a complex analysis. To determine which side of the mouth isbeing brushed at a particular time, the gyroscope data is scanned todetermine when the rotational orientation is greater than 180 degrees,indicating the left side, and when it is less than 180 degrees,indicating the right side. As explained above, top and bottom and gumbrushing information can also be obtained, without any calculations,simply by examining the data. The time sequence of data that is acquiredduring exercise and analyzed as discussed above can be used in a widevariety of ways.

In one embodiment, the accelerometers distinguish between lying down andeach upright position of sitting and standing based on the continuousoutput of the 3D accelerometer. The system can detect (a) extended timein a single position; (b) extended time sitting in a slouching posture(kyphosis) as opposed to sitting in an erect posture (lordosis); and (c)repetitive stressful movements, such as may be found on somemanufacturing lines, while typing for an extended period of time withoutproper wrist support, or while working all day at a weight liftingexercise, among others. In one alternative embodiment, angular positionsensors, one on each side of the hip joint, can be used to distinguishlying down, sitting, and standing positions. In another embodiment, thesystem repeatedly records position and/or posture data over time. In oneembodiment, magnetometers can be attached to a thigh and the torso toprovide absolute rotational position about an axis coincident withEarth's gravity vector (compass heading, or yaw). In another embodiment,the rotational position can be determined through the in-doorpositioning system as discussed above.

To improve a golf swing, the complex motion of the body first startswith the stance. The system checks that the golfer has a low center ofgravity to remain balanced throughout the swing path. The swing startswith the arms moving back in a straight line. When the club head reachesthe level of the hip, two things happen: there is a stern wrist cockthat acts as a hinge along with the left knee (for a right handedswing), building up its torque by moving into the same line as the bellybutton before the start of the upswing. As the swing continues to thetop of the backswing (again for right handed golf swing), the golfer'sleft arm should be perfectly straight and his right arm should be hingedat the elbow. The downswing begins with the hips and the lower bodyrather than the arms and upper body, with emphasis on the wrist cock. Asthe golfer's hips turn into the shot, the right elbow will drop straightdown, hugging the right side of the golfer's torso. As the right elbowdrops, the wrists begin to snap through from the wrist cock in thebackswing. A solid extension of the arms and good transfer of bodyshould put the golfer leaning up on his right toe, balanced, with thegolf club resting on the back of the golfers neck. Importantly, all ofthe movements occur with precise timing, while the head remainscompletely still with eyes focused on the ball throughout the entireswing.

The system can identify illnesses and prevent overexertion leading toillnesses such as a stroke. Depending on the severity of the stroke,enthusiasts can experience a loss of consciousness, cognitive deficits,speech dysfunction, limb weakness, hemiplegia, vertigo, diplopia, lowercranial nerve dysfunction, gaze deviation, ataxia, hemianopia, andaphasia, among others. Four classic syndromes that arecharacteristically caused by lacunar-type stroke are: pure motorhemiparesis, pure sensory syndrome, ataxic hemiparesis syndrome, andclumsy-hand dysarthria syndrome. enthusiasts with pure motor hemiparesispresent with face, arm, and leg weakness. This condition usually affectsthe extremities equally, but in some cases it affects one extremity morethan the other. The most common stroke location in affected enthusiastsis the posterior limb of the internal capsule, which carries thedescending corticospinal and corticobulbar fibers. Other strokelocations include the pons, midbrain, and medulla. Pure sensory syndromeis characterized by hemibody sensory symptoms that involve the face,arm, leg, and trunk. It is usually the result of an infarct in thethalamus. Ataxic hemiparesis syndrome features a combination ofcerebellar and motor symptoms on the same side of the body. The leg istypically more affected than the arm. This syndrome can occur as aresult of a stroke in the pons, the internal capsule, or the midbrain,or in the anterior cerebral artery distribution. enthusiasts withclumsy-hand dysarthria syndrome experience unilateral hand weakness anddysarthria. The dysarthria is often severe, whereas the hand involvementis more subtle, and enthusiasts may describe their hand movements as“awkward.” This syndrome is usually caused by an infarct in the pons.Different patterns of signs can provide clues as to both the locationand the mechanism of a particular stroke. The system can detect symptomssuggestive of a brainstem stroke include vertigo, diplopia, bilateralabnormalities, lower cranial nerve dysfunction, gaze deviation (towardthe side of weakness), and ataxia. Indications of higher corticaldysfunction-such as neglect, hemianopsia, aphasia, and gaze preference(opposite the side of weakness)-suggest hemispheric dysfunction withinvolvement of a superficial territory from an atherothrombotic orembolic occlusion of a mainstem vessel or peripheral branch.

To detect muscle weakness or numbness, in one embodiment, the systemapplies a pattern recognizer such as a neural network or a Hidden MarkovModel (HMM) to analyze accelerometer output. In another embodiment,electromyography (EMG) is used to detect muscle weakness. In anotherembodiment, EMG and a pattern analyzer is used to detect muscleweakness. In yet another embodiment, a pattern analyzer analyzes bothaccelerometer and EMG data to determine muscle weakness. In a furtherembodiment, historical ambulatory information (time and place) is usedto further detect changes in muscle strength. In yet other embodiments,accelerometer data is used to confirm that the enthusiast is at rest sothat EMG data can be accurately captured or to compensate for motionartifacts in the EMG data in accordance with a linear or non-linearcompensation table. In yet another embodiment, the EMG data is used todetect muscle fatigue and to generate a warning to the enthusiast to getto a resting place or a notification to a nurse or caregiver to rendertimely assistance. The amplitude of the EMG signal is stochastic(random) in nature and can be reasonably represented by a Gausiandistribution function. The amplitude of the signal can range from 0 to10 mV (peak-to-peak) or 0 to 1.5 mV (rms). The usable energy of thesignal is limited to the 0 to 500 Hz frequency range, with the dominantenergy being in the 50-150 Hz range. Usable signals are those withenergy above the electrical noise level. The dominant concern for theambient noise arises from the 60 Hz (or 50 Hz) radiation from powersources. The ambient noise signal may have an amplitude that is one tothree orders of magnitude greater than the EMG signal. There are twomain sources of motion artifact: one from the interface between thedetection surface of the electrode and the skin, the other from movementof the cable connecting the electrode to the amplifier. The electricalsignals of both noise sources have most of their energy in the frequencyrange from 0 to 20 Hz and can be reduced.

In one embodiment, the camera captures facial expression and a code suchas the Microsoft Emotion API takes a facial expression in an image as aninput, and returns the confidence across a set of emotions for each facein the image, as well as bounding box for the face, using the Face API.The emotions detected are anger, contempt, disgust, fear, happiness,neutral, sadness, and surprise. These emotions are understood to becross-culturally and universally communicated with particular facialexpressions. Alternatively, a marker for emotional arousal is galvanicskin response (GSR), also referred to as skin conductance (SC) orelectro-dermal activity (EDA). EDA modulates the amount of sweatsecretion from sweat glands. The amount of sweat glands varies acrossthe human body, being highest in hand and foot regions (200-600 sweatglands per cm2). While sweat secretion plays a major role forthermoregulation and sensory discrimination, changes in skin conductancein hand and foot regions are also triggered quite impressively byemotional stimulation: the higher the arousal, the higher the skinconductance. It is noteworthy to mention that both positive (“happy” or“joyful”) and negative (“threatening” or “saddening”) stimuli can resultin an increase in arousal—and in an increase in skin conductance. Skinconductance is not under conscious control. Instead, it is modulatedautonomously by sympathetic activity which drives human behavior,cognitive and emotional states on a subconscious level. Skin conductancetherefore offers direct insights into autonomous emotional regulation.It can be used as alternative to self-reflective test procedures,or—even better—as additional source of insight to validate verbalself-reports or interviews of a respondent. Based on the detectedemotion, the exercise can be increased, decreased, or stoppedaltogether.

Features of the auto-detection of exercise include the following:

-   -   1. An exercise system, comprising:        -   a processor running the motion analyzer and coupled to a            wireless transceiver;        -   an accelerometer coupled to the processor; and        -   a kinematic motion analysis module executed by the processor            to detect exercise type.    -   2. The system of claim 1, comprising a plurality of smart        modules mounted on an exerciser forming a mesh network.    -   3. The system of claim 1 where the electronic components,        sensors, and interconnects of the system monitor, record,        process and/or transmit events of interest (such as        accelerometers and gyroscopes for impact events, temperature        sensors for temperature and/or temperature gradients, pressure        sensors, moisture sensors, chemical sensors).    -   4. The system of claim 1 comprised for sensing and/or monitoring        impact events where the sensors are accelerometers, gyroscopes,        and/or pressure sensors.    -   5. The system of claim 1 comprised for sensing and/or monitoring        and/or controlling ongoing events where the sensors monitor        temperature, temperature gradients, motion, position,        environmental or chemical levels, or other such information.    -   6. The system of claim 1 comprised for sensing events or other        information including mounting multiple distributed sensors for        obtaining spatial and/or temporal distribution in the data        and/or multiple sensors sensing different information and data.    -   7. The system of claim 1 comprising a camera and an image        recognition module to determine kinematic movement.    -   8. The system of claim 1 including a statistical recognizer to        determine kinematic movement.    -   9. The system of claim 8, comprising a model-state that contains        the extracted features of body signatures and other associated        characteristics of body signatures.    -   10. The system of claim 1 comprising links connecting a root        node (torso) with connected joint, left shoulder (LS), right        shoulder (RS), left hip (LH), and right hip (RH), and left elbow        (LE), right elbow (RE), left knee (LK), and right knee (RK)        connect upper and lower extremities.    -   11. The system of claim 1 comprising a posture detection module.    -   12. The system of claim 1, comprising a module to detect a lying        down state and a standing state.    -   13. The system of claim 1, comprising a hidden markov model        module to detect muscle movement and exercise pattern.    -   14. The system of claim 1 comprising optimizing tennis shots to        improve serve, groundstroke, volley, half volley, smash,        forehand, backhand, flat, side spin, block, slice, topspin shot,        lob, passing shot, dropshot, cross-court shot, down-the-line        shot.    -   15. The system of claim 1, comprising an electromyography (EMG)        sensor to detect muscle strength or weakness.    -   16. The system of claim 1, comprising an emotion detector        wherein an exercise can be increased, decreased, or stopped        based on detected emotion.    -   17. The system of claim 17, wherein the detector comprises video        detection of faces or a GSR sensor.    -   18. The system of claim 1 comprising a cloud storage to receive        sensor data.    -   19. The system of claim 1, comprising a golf training module        that checks that a golfer has a low center of gravity to remain        balanced throughout a swing path, that a swing starts with the        arms moving back in a straight line, and when a club head        reaches the level of the hip, a wrist cock acts as a hinge along        with the left knee (for a right handed swing), building up        torque by moving into the same line as the belly button before        the start of the upswing. As the swing continues to the top of        the backswing (again for right handed golf swing), the golfer's        left arm is straight and a right arm is hinged at the elbow.    -   20. The system of claim 19, wherein the golf training module        checks that a downswing begins with the hips and the lower body        and as the golfer's hips turn into the shot, the right elbow        drops down, hugging the right side of the golfer's torso and        wrists begin to snap through from the wrist cock in the        backswing.    -   21. The system of claim 1, comprising a soccer training module        with kinematics of ball control, dribbling, passing, crossing,        shooting, heading, volleying, taking throw-ins, penalties,        corner kicks and free kicks, tackling, marking, juggling,        receiving, shielding, clearing, and goalkeeping.    -   22. The system of claim 1, comprising a basketball training        module with kinematics of crossover dribble, behind back, pull        back dribble, low dribble, basic dribble, between legs dribble,        Overhead Pass, Chest Pass, Push Pass, Baseball Pass,        Off-the-Dribble Pass, Bounce Pass, Jump Shot, Dunk, Free throw,        Layup, Three-Point Shot, Hook Shot.    -   23. The system of claim 1, comprising a baseball training module        with kinematics of Hitting, Bunting, Base Running and Stealing,        Sliding, Throwing, Fielding Ground Balls, Fielding Fly Balls,        Double Plays and Relays, Pitching and Catching, Changing Speeds,        Holding Runners, Pitching and Pitcher Fielding Plays, Catching        and Catcher Fielding Plays.

Data from multiple exercise sessions may be collected and used tocompile a history of the user's habits over an extended period of time,enabling the user's trainer to better understand user compliance issues.The trainer can review the data with the user and view the animations ofthe user's exercise sessions during an office visit, allowing thetrainer to better instruct the user in proper brushing technique. Thetrainer can also review the patient's brushing history over time, todetermine whether the patient's exercise technique is improving.

The sensor 14 can be integrated into objects already associated with thesporting activity. In one aspect, the sensing unit is integrated intothe ski boot or other boot. In another aspect, the sensing unit isintegrated into the binding for a ski boot or snowboarder boot. In stillanother aspect, the sensing unit is integrated into a ski, snowboard,mountain bike, windsurfer, windsurfer mast, roller blade boot,skate-board, kayak, or other vehicle. Collectively, the objects such asthe ski boot and the variety of vehicles are denoted as “implements”.Accordingly, when the sensing unit is not “stand alone”, the housingwhich integrates the controller subsystem with one or more sensors andbattery can be made from the material of the associated implement, inwhole or in part, such that the sensing unit becomes integral with theimplement. The universal interface is therefore not desired in thisaspect.

One skilled in the art will appreciate that, for this and otherprocesses and methods disclosed herein, the functions performed in theprocesses and methods may be implemented in differing order.Furthermore, the outlined steps and operations are only provided asexamples, and some of the steps and operations may be optional, combinedinto fewer steps and operations, or expanded into additional steps andoperations without detracting from the essence of the disclosedembodiments.

The embodiments described herein may include the use of a specialpurpose or general-purpose computer including various computer hardwareor software modules, as discussed in greater detail below.

Embodiments described herein may be implemented using computer-readablemedia for carrying or having computer-executable instructions or datastructures stored thereon. Such computer-readable media may be anyavailable media that may be accessed by a general purpose or specialpurpose computer. By way of example, and not limitation, suchcomputer-readable media may include tangible computer-readable storagemedia including RAM, ROM, EEPROM, CD-ROM or other optical disk storage,magnetic disk storage or other magnetic storage devices, or any otherstorage medium which may be used to carry or store desired program codein the form of computer-executable instructions or data structures andwhich may be accessed by a general purpose or special purpose computer.Combinations of the above may also be included within the scope ofcomputer-readable media. Computer-executable instructions comprise, forexample, instructions and data which cause a general purpose computer,special purpose computer, or special purpose processing device toperform a certain function or group of functions. Although the subjectmatter has been described in language specific to structural featuresand/or methodological acts, it is to be understood that the subjectmatter defined in the appended claims is not necessarily limited to thespecific features or acts described above. Rather, the specific featuresand acts described above are disclosed as example forms of implementingthe claims. As used herein, the term “module” or “component” may referto software objects or routines that execute on the computing system.The different components, modules, engines, and services describedherein may be implemented as objects or processes that execute on thecomputing system (e.g., as separate threads). While the system andmethods described herein may be preferably implemented in software,implementations in hardware or a combination of software and hardwareare also possible and contemplated. In this description, a “computingentity” may be any computing system as previously defined herein, or anymodule or combination of modulates running on a computing system. Allexamples and conditional language recited herein are intended forpedagogical objects to aid the reader in understanding the invention andthe concepts contributed by the inventor to furthering the art, and areto be construed as being without limitation to such specifically recitedexamples and conditions. Although embodiments of the present inventionshave been described in detail, it should be understood that the variouschanges, substitutions, and alterations could be made hereto withoutdeparting from the spirit and scope of the invention.

What is claimed is:
 1. A system, comprising: a processor coupled to awireless transceiver; a camera to capture device environment, whereinthe camera captures distances of the neighboring objects from thecamera; code to change insurance cost based on trip distance and riskassociated with trip distance.
 2. The system of claim 1, comprising apressure sensor, a motion sensor, a finger sensor, a digit motionsensor, an EKG sensor, a bio sensor, a medical sensor, or a temperaturesensor.
 3. The system of claim 1, comprising a module to follow otherthird parties or other devices in a flock.
 4. The system of claim 1,wherein the device comprises a road contacting surface, air contactingsurface or water contacting surface.
 5. The system of claim 1,comprising a gesture identifying component configured to identify handor finger gesture to the device.
 6. The system of claim 1, comprising anoptical positioning system.
 7. The system of claim 6, wherein thepositioning system comprises a laser positioning system.
 8. The systemof claim 1, comprising a sensor to detect an imminent impact andactivate crash protection for a user.
 9. The system of claim 1,comprising a hidden markov model (HMM) or a deep learning machine todetect patterns from the camera and the accelerometer.
 10. The system ofclaim 1, comprising an emotion detector to detect a user condition. 11.The system of claim 1, comprising an accelerometer and a camera todetect a potential impact.
 12. The system of claim 1, comprising a cloudcomputer coupled to the processor to predict an impact and to protect auser for the impact.
 13. The system of claim 1, comprising at least onesensor selected from a sensor set comprising: a pressure sensor, amotion sensor, a finger sensor, a digit motion sensor, an EKG sensor, abio sensor, a medical sensor, a temperature sensor.
 14. The system ofclaim 1, comprising a module to follow other third parties or otherdevices in a flock.
 15. The system of claim 1, wherein the devicecomprises a road contacting surface, further comprising a positioningsystem.
 16. The system of claim 1, comprising a gesture identifyingcomponent to recognize hand or finger gesture to the device.
 17. Thesystem of claim 1, comprising returning from user control to processorcontrol upon user activation.
 18. The system of claim 1, wherein priorto taking control, the processor scans the surroundings and determinesany obstacles or objects in the immediate vicinity which reduce safety19. The system of claim 1, wherein a driver maintains control of thedevice before entering into an autonomous mode.
 20. The system of claim1, comprising code to relinquish control of the device back to a driverif safe, and otherwise collect travel data from neighboring vehicles andcoordinate travel path for a plurality of vehicles to increase safetyand decrease travel cost, travel time, energy cost, air pollution.